{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ab0a6581-7245-4906-a830-09cf6e955144",
   "metadata": {
    "tags": []
   },
   "source": [
    "# <font face=\"仿宋\">课程说明："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac94a98-77ba-4d57-9781-1c93f3ebe9eb",
   "metadata": {},
   "source": [
    "- 体验课内容节选自[《2025大模型Agent智能体开发实战》](https://whakv.xetslk.com/s/3xFEAA)完整版付费课程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79dea34-b771-4da3-8537-dfeaff711405",
   "metadata": {},
   "source": [
    "&emsp;&emsp;体验课时间有限，若想深度学习大模型技术，欢迎大家报名由我主讲的[《2025大模型Agent智能体开发实战》](https://whakv.xetslk.com/s/3xFEAA)："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a66b004-e72d-4fc1-8b49-d085dd6bd16e",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/a4f3bdfb511aab72c83ee7d1fbefeff.png\" alt=\"a4f3bdfb511aab72c83ee7d1fbefeff\" style=\"zoom:15%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a157df6-1d3b-4cac-9346-2b8d7fe09626",
   "metadata": {},
   "source": [
    "**[《2025大模型Agent智能体开发实战》](https://whakv.xetslk.com/s/3xFEAA)为【100+小时】体系大课，总共20大模块精讲精析，零基础直达大模型企业级应用！**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1657a9db-baf9-4cbc-97c6-82c0eca65a37",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/bbf269a0de3abba3e4ee21b29bc7cfc.png\" alt=\"bbf269a0de3abba3e4ee21b29bc7cfc\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aacb8177-a0de-4a62-a5b2-252444d9bf28",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/eb29b5d74f0b1d86aea1b5698f6ea9d.png\" alt=\"eb29b5d74f0b1d86aea1b5698f6ea9d\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b429078-7812-4fdd-aa86-58a4271927d7",
   "metadata": {},
   "source": [
    "**[《2025大模型Agent智能体开发实战》](https://whakv.xetslk.com/s/3xFEAA)2025年新春班上新&新年双重特惠，立减2000，九天老师直播间专属优惠券1000，仅需2999即可入学，【仅限前10名】<span style=\"color:red;\">详细信息扫码添加助教，回复“大模型”，即可领取课程大纲&查看课程详情👇</span>**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "613f8381-e87c-4f72-954d-556ec18af4d3",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/c6847a817fd7efd0cddcb1bcac217c3.png\" alt=\"c6847a817fd7efd0cddcb1bcac217c3\" style=\"zoom:55%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f6d1ee-17a1-4b5e-8284-f126e3572f58",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250107200452887.png\" alt=\"image-20250107200452887\" style=\"zoom:59%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ede326-bca2-4ada-8724-c3aace5fb89a",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250107200502389.png\" alt=\"image-20250107200502389\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea865fe1-b6dd-4cea-8b5b-220f13c0c99d",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c13457-61dd-4655-b1fc-94eec6951e29",
   "metadata": {},
   "source": [
    "# <center> 《2025大模型智能体Agent开发实战》体验课   \n",
    "## <center> DeepSeek-v3企业知识库问答入门实战\n",
    "## <center> 借助GraphRAG搭建知识库问答机器人"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e350bba-9b0c-49e9-a930-950af6778ca1",
   "metadata": {},
   "source": [
    "### 一、DeepSeek-v3大模型介绍与微软GraphRAG项目介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28ac110-296a-4776-b469-a64a5e8c9bc2",
   "metadata": {},
   "source": [
    "#### 1.国产开源大模型之光：DeepSeek v3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5480f35d-49e5-4ec2-bf87-4d547acc18df",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108182903085.png\" alt=\"image-20250108182903085\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d1196a5-83fc-4504-bbe3-0b3129e75866",
   "metadata": {},
   "source": [
    "&emsp;&emsp;DeepSeek-V3是由中国人工智能公司深度求索（DeepSeek）于2024年12月26日发布的开源大型语言模型。该模型采用混合专家（MoE）架构，拥有6710亿个总参数，每个token激活370亿参数，并在14.8万亿token上进行了预训练。在多项基准测试中，DeepSeek-V3表现出色，尤其在代码和数学任务上，超越了其他开源模型，其性能与领先的闭源模型相当。此外，模型的生成速度提升至每秒60个token，相比前一版本提高了3倍。值得注意的是，DeepSeek-V3的训练成本约为557.6万美元，显著低于同类模型，体现了高效的训练策略和资源利用。用户可以通过DeepSeek的官方网站与该模型进行交互，体验其强大的语言理解和生成能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c918b5f9-e315-40c3-aa75-c2efffbd533f",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108183202945.png\" alt=\"image-20250108183202945\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96a880d2-e135-421b-aa73-ab99e64bba5b",
   "metadata": {},
   "source": [
    "DeepSeek v3采用混合专家（MoE）架构，拥有6710亿个总参数，每个token激活370亿参数。 \n",
    "\n",
    "**模型架构与创新：**\n",
    "\n",
    "- **多头潜在注意力（MLA）：** 通过低秩压缩技术减少推理时的Key-Value缓存，提升推理效率。 \n",
    "\n",
    "- **DeepSeekMoE架构：** 采用细粒度的专家分配和共享专家机制，实现高效训练。 \n",
    "\n",
    "- **无辅助损失的负载均衡策略：** 通过动态调整专家偏置，确保专家负载均衡，提升模型性能。 \n",
    "\n",
    "- **多令牌预测（MTP）：** 支持多令牌预测，提高训练信号密度，并通过推测解码加速推理。 \n",
    "\n",
    "- **FP8混合精度训练：** 在大规模模型中验证了FP8训练的可行性，降低内存消耗和训练成本。 \n",
    "\n",
    "**训练与性能：**\n",
    "\n",
    "- **训练数据：** 在14.8万亿高质量token上进行预训练，模型训练过程稳定。 \n",
    "\n",
    "- **训练成本：** 总训练成本约为557.6万美元，显著低于同类模型。 \n",
    "\n",
    "- **性能表现：** 在多种基准测试中表现优异，尤其在代码和数学任务上，超越其他开源模型，与领先的闭源模型相当。 \n",
    "\n",
    "DeepSeek-V3的发布标志着开源模型在性能和效率上的新高度，为自然语言处理技术的进步提供了新的示例。 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22fad98d-495e-4953-abd3-2054793a93cd",
   "metadata": {},
   "source": [
    "#### 2.知识库问答最强检索增强技术：GraphRAG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f8a1bef-cfe7-43c8-9310-05713e749a1b",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/2bc296344f628da9922f47d501cb248.png\" alt=\"2bc296344f628da9922f47d501cb248\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2ffef9-cc96-4466-8b81-84e811b8fc2a",
   "metadata": {},
   "source": [
    "&emsp;&emsp;**检索增强生成（RAG）** 是一种通过结合真实世界的信息来提升大型语言模型（LLM）输出质量的技术。RAG 技术是大多数基于 LLM 的工具中的一个重要组成部分。大多数 RAG 方法使用 **向量相似性** 作为检索技术，我们将其称为 **基线 RAG（Baseline RAG）**。\n",
    "\n",
    "**GraphRAG** 则使用 **知识图谱** 来在推理复杂信息时显著提升问答性能。当需要对复杂数据进行推理时，GraphRAG 展示了优于基线 RAG 的性能，特别是在 **知识图谱** 的帮助下。\n",
    "\n",
    "&emsp;&emsp;RAG 技术在帮助 LLM 推理私有数据集方面显示了很大的潜力——例如，LLM 没有在训练时接触过的、企业的专有研究、业务文档或通信数据。基线 RAG 技术最初是为了解决这个问题而提出的，但我们观察到，在某些情况下，基线 RAG 的表现并不理想。以下是几个典型的场景：\n",
    "\n",
    "1. **基线 RAG 很难将信息串联起来**：当一个问题的答案需要通过多个不同的信息片段，并通过它们共享的属性来连接，进而提供新的综合见解时，基线 RAG 表现得很差。\n",
    "   \n",
    "   例如，在回答类似“如何通过现有的数据推断出新结论”这种问题时，基线 RAG 无法很好地处理这些散布在不同文档中的相关信息，它可能会遗漏一些关键联系点。\n",
    "\n",
    "2. **基线 RAG 无法有效理解大型数据集或单一大文档的整体语义概念**：当被要求在大量数据或复杂文档中进行总结、提炼和理解时，基线 RAG 往往表现不佳。\n",
    "\n",
    "   例如，如果问题要求对整个文档或多篇文档的主题进行总结和理解，基线 RAG 的简单向量检索方法可能无法处理文档间的复杂关系，导致对全局语义的理解不完整。\n",
    "\n",
    "&emsp;&emsp;为了应对这些挑战，技术社区正在努力开发扩展和增强 RAG 的方法。**微软研究院**（Microsoft Research）提出的 **GraphRAG** 方法，使用 **LLM** 基于输入语料库构建 **知识图谱**。这个图谱与社区总结和图谱机器学习输出结合，能够在查询时增强提示（prompt）。GraphRAG 在回答以上两类问题时，展示了 **显著的改进**，尤其是在 **复杂信息的推理能力** 和 **智能性** 上，超越了基线 RAG 之前应用于私有数据集的其他方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7c4b92-6920-46c7-9362-0b9fc61c1d79",
   "metadata": {},
   "source": [
    "#### 3.微软GraphRAG项目简介"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee08eaee-74e6-4d99-9d28-210628375252",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128154854230.png\" alt=\"image-20241128154854230\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efee369c-dc2f-4e06-b308-1f8050e6bee1",
   "metadata": {},
   "source": [
    "> 项目地址：https://github.com/microsoft/graphrag/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d06f3437-ac48-4c4a-84b9-99cd1a1ebd00",
   "metadata": {},
   "source": [
    "&emsp;&emsp;**GraphRAG** 是微软研究院开发的一种先进的增强检索生成（RAG）框架，旨在提升语言模型（LLM）在处理复杂数据时的性能。与传统的 RAG 方法依赖向量相似性检索不同，**GraphRAG** 利用 **知识图谱** 来显著增强语言模型的问答能力，特别是在处理私有数据集或大型、复杂数据集时表现尤为出色。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "378da838-ad92-4674-8c97-66e4acbe1c5b",
   "metadata": {},
   "source": [
    "#### 2.GraphRAG核心特点"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15e0da49-f049-423a-8584-e197c9fff367",
   "metadata": {},
   "source": [
    "&emsp;&emsp;传统的 **Baseline RAG** 方法在某些情况下表现不佳，尤其是当查询需要在不同信息片段之间建立联系时，或是当需要对大规模数据集进行整体理解时。GraphRAG 通过以下方式克服了这些问题：\n",
    "- **更好的连接信息点**：GraphRAG 能够处理那些需要从多个数据点合成新见解的任务。\n",
    "- **更全面的理解能力**：GraphRAG 更擅长对大型数据集进行全面理解，能够更好地处理复杂的抽象问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7b95071-887b-4bde-9e21-378bbf990c8a",
   "metadata": {},
   "source": [
    "&emsp;&emsp;而借助微软开源的GeaphRAG项目，我们可以快速做到以下事项：\n",
    "- **基于图的检索**：传统的 RAG 方法使用向量相似性进行检索，而 GraphRAG 引入了知识图谱来捕捉实体、关系及其他重要元数据，从而更有效地进行推理。\n",
    "- **层次聚类**：GraphRAG 使用 **Leiden** 技术进行层次聚类，将实体及其关系进行组织，提供更丰富的上下文信息来处理复杂的查询。\n",
    "- **多模式查询**：支持多种查询模式：\n",
    "  - **全局搜索**：通过利用社区总结来进行全局性推理。\n",
    "  - **局部搜索**：通过扩展相关实体的邻居和关联概念来进行具体实体的推理。\n",
    "  - **DRIFT 搜索**：结合局部搜索和社区信息，提供更准确和相关的答案。\n",
    "- **图机器学习**：集成了图机器学习技术，提升查询响应质量，并提供来自结构化和非结构化数据的深度洞察。\n",
    "- **Prompt 调优**：提供调优工具，帮助根据特定数据和需求调整查询提示，从而提高结果质量。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97dc183b-040e-4d2c-b53f-ca639a7b1276",
   "metadata": {},
   "source": [
    "#### 3.GraphRAG运行流程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691206d2-b8a2-4f7a-b4c2-f0842ff0c246",
   "metadata": {},
   "source": [
    " **索引（Indexing）过程**\n",
    "1. **文本单元切分**：将输入文本分割成 **TextUnits**，每个 TextUnit 是一个可分析的单元，用于提取关键信息。\n",
    "2. **实体和关系提取**：使用 LLM 从 TextUnits 中提取实体、关系和关键声明。\n",
    "3. **图构建**：构建知识图谱，使用 Leiden 算法进行实体的层次聚类。每个实体用节点表示，节点的大小和颜色分别代表实体的度数和所属社区。\n",
    "4. **社区总结**：从下到上生成每个社区及其成员的总结，帮助全局理解数据集。\n",
    "\n",
    " **查询（Query）过程**        \n",
    " 索引完成后，用户可以通过不同的搜索模式进行查询：\n",
    "- **全局搜索**：当我们想了解整个语料库或数据集的整体概况时，GraphRAG 可以利用 社区总结 来快速推理和获取信息。这种方式适用于大范围问题，如某个主题的总体理解。\n",
    "- **局部搜索**：如果问题关注于某个特定的实体，GraphRAG 会向该实体的 邻居（即相关实体）扩展搜索，以获得更详细和精准的答案。\n",
    "- **DRIFT 搜索**：这是对局部搜索的增强，除了获取邻居和相关概念，还引入了 社区信息 的上下文，从而提供更深入的推理和连接。\n",
    "\n",
    " **Prompt 调优**        \n",
    " 为了获得最佳性能，GraphRAG 强烈建议进行 **Prompt 调优**，确保模型可以根据你的特定数据和查询需求进行优化，从而提供更准确和相关的答案。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1d46c2-5735-4505-b40f-82400ea7010f",
   "metadata": {},
   "source": [
    "#### 4.GraphRAG核心原理回顾"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a1b3ba-ca82-4d78-b58a-58d246eab76a",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/GraphRAG%E5%AF%BC%E5%9B%BE.png\" alt=\"GraphRAG导图\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf46e6ca-ad4b-4d5c-84bf-41fa52784104",
   "metadata": {},
   "source": [
    "### 二、GraphRAG安装与Indexing检索流程实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93b114f6-fbd7-445b-9abe-bfbfeb82d058",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de083291-e30e-4058-af0c-b85550fa7a2a",
   "metadata": {},
   "source": [
    "> 注，以下实验环境均为Ubuntu系统，以更好的模拟真实企业应用场景，其中大多数方法也可以直接迁移至Windows操作系统中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84b7129f-258b-4149-87d1-15ce23543b07",
   "metadata": {},
   "source": [
    "#### <center> 【补充阅读】在线算力租赁平台AutoDL使用指南与虚拟环境创建方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "593feaad-693b-49ac-95a9-442c07448659",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "baba93db-acd4-4dba-9054-1c287adbbe6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('【补充材料】在线算力租赁平台AutoDL使用指南与虚拟环境创建方法.md', 'r') as file:\n",
    "    md_content = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "09b6999d-c7a9-4fe0-b437-8c57061056b3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "## 【补充材料】在线算力租赁平台AutoDL使用指南与虚拟环境创建方法\n",
       "\n",
       "### Step 1.AutoDL算力租赁流程\n",
       "\n",
       "- AutoDL简介\n",
       "\n",
       "​\tAutoDL是一个稳定高效的算力租赁平台，AutoDL支持一键部署基础Linux基础环境、镜像环境，同时支持镜像文件保存与迁移、跨实力读取文件等，且平台显卡较为充裕，并支持多种不同计费方式，对于大模型初学者来说是非常不错的算力租赁渠道。\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022181752210.png\" alt=\"image-20241022181752210\" style=\"zoom:33%;\" />\n",
       "\n",
       "但与此同时，AutoDL也有一定的局限，如没有公网IP、不支持Docker镜像等。不过对于本次公开课，对于不具备本地硬件环境的同学来说，AutoDL肯定是最佳算力租赁渠道，公开课所有的演示流程也将是基于AutoDL环境的操作，便于各位同学跟着公开课内容逐步手动实现。\n",
       "\n",
       "- AutoDL注册与充值\n",
       "\n",
       "​\t点击AutoDL官网即可注册与登录👉https://www.autodl.com/login\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022181517221.png\" alt=\"image-20241022181517221\" style=\"zoom:33%;\" />\n",
       "\n",
       "除了常规登录流程外，AutoDL还要求绑定微信和进行实名认证。实名认证其实是目前所有算力租赁平台的基本要求，大家按照AutoDL官网指引👉https://www.autodl.com/console/center/account/auth，一步步操作即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022182150411.png\" alt=\"image-20241022182150411\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 账户充值\n",
       "\n",
       "​\tAutoDL采用的是预付费模式，即需要先在AutoDL平台上储值，然后再租赁相关算力。AutoDL充值地址👉https://www.autodl.com/recharge。需要注意的是，若需要全程完成公开课的模型训练流程，则需要约50元左右算力成本，而若只想简单测试最终模型效果，则算力成本在10元以内，大家可以根据自己实际需求进行充值。\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022182525178.png\" alt=\"image-20241022182525178\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 算力租赁\n",
       "\n",
       "​\t接下来就需要进行GPU租赁了，本次公开课实验需要28G-30G左右显存，同时为了给大家介绍单机多卡的模型训练方法，推荐租赁双卡3090服务器，相对来说会较为划算。此外，也可以考虑租赁更高配置的服务器，如双卡4090、双卡L20、双卡A100等，可根据实际需求进行租赁。AutoDL算力市场👉：https://www.autodl.com/market/list\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022182739362.png\" alt=\"image-20241022182739362\" style=\"zoom:80%;\" />\n",
       "\n",
       "这里以租赁双卡3090为例进行介绍，计费方式可选根据实际需求进行选择，然后点击3090专区，并GPU数量选择2，然后任意选择一个有闲置GPU的服务器，点击2卡可租即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022182943344.png\" alt=\"image-20241022182943344\" style=\"zoom:33%;\" />\n",
       "\n",
       "然后再次确认计费方式，以及账户剩余费用是否足够支付，在基础镜像中，选择一个镜像：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183212923.png\" alt=\"image-20241022183212923\" style=\"zoom:33%;\" />\n",
       "\n",
       "这里我们选择Pytorch 2.3.0（即最高版本）+Python 3.12+CUDA 12.1\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183244056.png\" alt=\"image-20241022183244056\" style=\"zoom:33%;\" />\n",
       "\n",
       "选择完毕后，点击立即创建即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183035745.png\" alt=\"image-20241022183035745\" style=\"zoom:33%;\" />\n",
       "\n",
       "> 注，这里的镜像配置和计费方式，都可以随时修改。\n",
       "\n",
       "- 查看当前服务器\n",
       "\n",
       "​\t创建完成后，页面会自动跳转到当前账号的在线服务器管理页面，这里可以看到刚刚租赁的服务器正在开机中：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183434334.png\" alt=\"image-20241022183434334\" style=\"zoom:33%;\" />\n",
       "\n",
       "稍作等待，待显示`运行中`时，代表远程服务器已经顺利运行，接下来则需要考虑如何连接远程服务器。\n",
       "\n",
       "### Step 2.使用FinalShell连接远程服务器\n",
       "\n",
       "​\t连接远程服务器的方式有很多种，首先服务器已预安装了JupyterLab，我们可以直接通过JupyterLab中的Terminal来连接服务器，并通过命令行的方式操作远程服务器。但由于JupyterLab中Termianl界面较为原始，对新人用户操作并不友好。因此推荐使用专门的远程终端连接软件连接远程服务器，这里推荐使用FinalShell，**下图扫码添加英英助教，回复LLM**，即可领取本次公开课课件：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/fe62e9ddf03d5a6cfc821ed2ba905f2.png\" style=\"zoom: 50%;\" />\n",
       "\n",
       "其中finalshell软件就在本次公开课的课件中：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022184011341.png\" alt=\"image-20241022184011341\" style=\"zoom:33%;\" />\n",
       "\n",
       "下载后按照普通软件安装流程进行安装即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220206041.png\" alt=\"image-20240423220206041\" style=\"zoom:33%;\" />\n",
       "\n",
       "安装完成后，即可使用FinalShell连接远程服务器。由于接下来FinalShell连接远程服务器需要使用远程服务器的地址和端口，因此我们需要先查阅此时远程服务器的基本信息。回到AutoDL控制台页面，分别复制登陆指令和密码到任意文本编辑器中：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220600875.png\" alt=\"image-20240423220600875\" style=\"zoom:33%;\" />\n",
       "\n",
       "例如此时改服务器的登录指令为（需要找到你的主机的登录指令和密码）：\n",
       "\n",
       "```bash\n",
       "ssh -p 24503 root@conxxx\n",
       "```\n",
       "\n",
       "密码为：\n",
       "\n",
       "```bash\n",
       "wxxxx\n",
       "```\n",
       "\n",
       "记录好了之后即可打开FincalShell，使用该信息连接远程服务器。接下来打开FinalShell：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220233458.png\" alt=\"image-20240423220233458\" style=\"zoom: 50%;\" />\n",
       "\n",
       "首次使用时点击左上方文件夹，进入连接管理器：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220319086.png\" alt=\"image-20240423220319086\" style=\"zoom:33%;\" />\n",
       "\n",
       "再点击创建新的连接，并在弹出的选项中选择SSH连接（Linux）：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220420227.png\" alt=\"image-20240423220420227\" style=\"zoom:33%;\" />\n",
       "\n",
       "此处名称可以随意填写，主机为`conxxx`，也就是登陆指令root@conxxx中的conxxx部分，端口为24503，也就是登录指令中ssh -p 24503 root@conxxx的24503部分，用户名为root，也就是root@conxxx中的conxxx中的root，而密码则是wxxxx，也就是此前复制的密码：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423220958263.png\" alt=\"image-20240423220958263\" style=\"zoom:33%;\" />\n",
       "\n",
       "创建完成后回到连接管理器页面，双击刚刚创建的连接：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183705711.png\" alt=\"image-20241022183705711\" style=\"zoom: 33%;\" />\n",
       "\n",
       "即可进入到连接页面，此时点击接受并保存即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423221424622.png\" alt=\"image-20240423221424622\" style=\"zoom:33%;\" />\n",
       "\n",
       "当进入到如下页面，则说明已经连接成功：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423221510561.png\" alt=\"image-20240423221510561\" style=\"zoom:33%;\" />\n",
       "\n",
       "接下来即可借助FinalShell的命令行来操作远程服务器，在远程服务器上进行模型训练相关操作。\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/fe62e9ddf03d5a6cfc821ed2ba905f2.png\" alt=\"fe62e9ddf03d5a6cfc821ed2ba905f2\" style=\"zoom:33%;\" />\n",
       "\n",
       "### Step 3.创建虚拟环境\n",
       "\n",
       "- 打开finalshell，创建和服务器的shell连接\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022183705711.png\" alt=\"image-20241022183705711\" style=\"zoom: 33%;\" />\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20240423221510561.png\" alt=\"image-20240423221510561\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 创建MateConv虚拟环境（虚拟环境名称可自定义）\n",
       "\n",
       "```bash\n",
       "conda create --name MateConv python=3.10\n",
       "conda init\n",
       "source ~/.bashrc\n",
       "conda activate MateConv\n",
       "```\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022185015223.png\" alt=\"image-20241022185015223\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 创建Jupyter Kernel\n",
       "\n",
       "```Bash\n",
       "conda install jupyterlab\n",
       "conda install ipykernel\n",
       "python -m ipykernel install --user --name MateConv --display-name \"Python (MateConv)\"\n",
       "```\n",
       "\n",
       "- 创建项目主目录\n",
       "\n",
       "```bash\n",
       " cd ~/autodl-tmp/\n",
       " mkdir MateConv\n",
       "```\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022185502717.png\" alt=\"image-20241022185502717\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 打开Jupyter\n",
       "\n",
       "```bash\n",
       "cd ~/autodl-tmp/MateConv\n",
       "juputer lab --allow-root\n",
       "```\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191004793.png\" alt=\"image-20241022191004793\" style=\"zoom:33%;\" />\n",
       "\n",
       "- 本地连接Jupyter并查看虚拟环境kernel\n",
       "\n",
       "​\t当服务器开启Jupyter时，Windows环境下，本地可通过AutoDL-SSH-Tools进行访问，**下图扫码添加英英助教，回复LLM**，即可领取本次公开课课件：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/fe62e9ddf03d5a6cfc821ed2ba905f2.png\" style=\"zoom: 50%;\" />\n",
       "\n",
       "其中AutoDL-SSH-Tools软件就在本次公开课的课件中：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191251924.png\" alt=\"image-20241022191251924\" style=\"zoom:33%;\" />\n",
       "\n",
       "下载完成后解压缩后双击打开AutoDL.exe\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191309818.png\" alt=\"image-20241022191309818\" style=\"zoom:33%;\" />\n",
       "\n",
       "在弹出的栏页中，SSH指令和密码就是AutoDL官网上给每个实例提供的密钥：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191336574.png\" alt=\"image-20241022191336574\" style=\"zoom:33%;\" />\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191413221.png\" alt=\"image-20241022191413221\" style=\"zoom:33%;\" />\n",
       "\n",
       "代理端口填写8889（也就是Juypter运行端口），点击开始代理，然后点击访问即可：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191503555.png\" alt=\"image-20241022191503555\" style=\"zoom:33%;\" />\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191655526.png\" alt=\"image-20241022191655526\" style=\"zoom:50%;\" />\n",
       "\n",
       "然后任意打开一个Kernel，选择找到刚才安装的虚拟环境即可\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191555056.png\" alt=\"image-20241022191555056\" style=\"zoom:33%;\" />\n",
       "\n",
       "> Mac登陆指令，打开终端并输入：（以下要填入自己的URL和port端口号）\n",
       ">\n",
       "> ssh -CNg -L 8889:127.0.0.1:8889  root@YOUR_URL -p port\n",
       ">\n",
       "> 然后根据提示输入密码\n",
       "\n",
       "接下来再次点击finalshell中+号，再次创建一个和服务器的连接：\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191853620.png\" alt=\"image-20241022191853620\" style=\"zoom:33%;\" />\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191914659.png\" alt=\"image-20241022191914659\" style=\"zoom:33%;\" />\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022191928536.png\" alt=\"image-20241022191928536\" style=\"zoom:33%;\" />\n",
       "\n",
       "然后再次进入到虚拟环境，准备后续的操作\n",
       "\n",
       "```bash\n",
       "conda activate MateConv\n",
       "```\n",
       "\n",
       "<img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241022192027587.png\" alt=\"image-20241022192027587\" style=\"zoom:33%;\" />"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(md_content))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe6eb76e-5580-4c02-ab74-95a42d606e24",
   "metadata": {},
   "source": [
    "《AutoDL公开课》：https://www.bilibili.com/video/BV1bxB7YYEST"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8282982-f4ea-456e-8f26-97738523423c",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108184316832.png\" alt=\"image-20250108184316832\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "470962f5-555d-4c69-8941-541e9bacb7a6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed9f1b79-e753-48d3-a2b3-a54c159bc0ed",
   "metadata": {},
   "source": [
    "#### 1.GraphRAG安装与项目创建"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b77e0ba-7e8c-4105-9175-9c18480c40c4",
   "metadata": {},
   "source": [
    "- **Step 1.使用pip安装graphrag**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a80b7251-1292-4d25-861a-8331bc15f40b",
   "metadata": {},
   "source": [
    "```bash\n",
    "pip install graphrag\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f735a910-c1b2-436b-89df-a5eb8a1df979",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
      "Requirement already satisfied: graphrag in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (0.5.0)\n",
      "Requirement already satisfied: aiofiles<25.0.0,>=24.1.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (24.1.0)\n",
      "Requirement already satisfied: aiolimiter<2.0.0,>=1.1.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.1.0)\n",
      "Requirement already satisfied: azure-identity<2.0.0,>=1.17.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.19.0)\n",
      "Requirement already satisfied: azure-search-documents<12.0.0,>=11.4.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (11.5.2)\n",
      "Requirement already satisfied: azure-storage-blob<13.0.0,>=12.22.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (12.24.0)\n",
      "Requirement already satisfied: datashaper<0.0.50,>=0.0.49 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.0.49)\n",
      "Requirement already satisfied: devtools<0.13.0,>=0.12.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.12.2)\n",
      "Requirement already satisfied: environs<12.0.0,>=11.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (11.2.1)\n",
      "Requirement already satisfied: future<2.0.0,>=1.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.0.0)\n",
      "Requirement already satisfied: graspologic<4.0.0,>=3.4.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (3.4.1)\n",
      "Requirement already satisfied: json-repair<0.31.0,>=0.30.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.30.2)\n",
      "Requirement already satisfied: lancedb<0.14.0,>=0.13.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.13.0)\n",
      "Requirement already satisfied: matplotlib<4.0.0,>=3.9.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (3.9.2)\n",
      "Requirement already satisfied: networkx<4,>=3 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (3.4.2)\n",
      "Requirement already satisfied: nltk==3.9.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (3.9.1)\n",
      "Requirement already satisfied: numpy<2.0.0,>=1.25.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.26.4)\n",
      "Requirement already satisfied: openai<2.0.0,>=1.51.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.55.1)\n",
      "Requirement already satisfied: pandas<3.0.0,>=2.2.3 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (2.2.3)\n",
      "Requirement already satisfied: pyaml-env<2.0.0,>=1.2.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.2.1)\n",
      "Requirement already satisfied: pyarrow<16.0.0,>=15.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (15.0.2)\n",
      "Requirement already satisfied: pydantic<3.0.0,>=2.9.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (2.10.1)\n",
      "Requirement already satisfied: python-dotenv<2.0.0,>=1.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (1.0.1)\n",
      "Requirement already satisfied: pyyaml<7.0.0,>=6.0.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (6.0.2)\n",
      "Requirement already satisfied: rich<14.0.0,>=13.6.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (13.9.4)\n",
      "Requirement already satisfied: tenacity<10.0.0,>=9.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (9.0.0)\n",
      "Requirement already satisfied: tiktoken<0.8.0,>=0.7.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.7.0)\n",
      "Requirement already satisfied: typer<0.13.0,>=0.12.5 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.12.5)\n",
      "Requirement already satisfied: typing-extensions<5.0.0,>=4.12.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (4.12.2)\n",
      "Requirement already satisfied: umap-learn<0.6.0,>=0.5.6 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graphrag) (0.5.7)\n",
      "Requirement already satisfied: click in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from nltk==3.9.1->graphrag) (8.1.7)\n",
      "Requirement already satisfied: joblib in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from nltk==3.9.1->graphrag) (1.4.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from nltk==3.9.1->graphrag) (2024.11.6)\n",
      "Requirement already satisfied: tqdm in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from nltk==3.9.1->graphrag) (4.67.1)\n",
      "Requirement already satisfied: azure-core>=1.31.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag) (1.32.0)\n",
      "Requirement already satisfied: cryptography>=2.5 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag) (43.0.3)\n",
      "Requirement already satisfied: msal>=1.30.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag) (1.31.1)\n",
      "Requirement already satisfied: msal-extensions>=1.2.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-identity<2.0.0,>=1.17.1->graphrag) (1.2.0)\n",
      "Requirement already satisfied: azure-common>=1.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-search-documents<12.0.0,>=11.4.0->graphrag) (1.1.28)\n",
      "Requirement already satisfied: isodate>=0.6.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from azure-search-documents<12.0.0,>=11.4.0->graphrag) (0.7.2)\n",
      "Requirement already satisfied: diskcache<6.0.0,>=5.6.3 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from datashaper<0.0.50,>=0.0.49->graphrag) (5.6.3)\n",
      "Requirement already satisfied: jsonschema<5.0.0,>=4.21.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from datashaper<0.0.50,>=0.0.49->graphrag) (4.23.0)\n",
      "Requirement already satisfied: asttokens<3.0.0,>=2.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from devtools<0.13.0,>=0.12.2->graphrag) (2.0.5)\n",
      "Requirement already satisfied: executing>=1.1.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from devtools<0.13.0,>=0.12.2->graphrag) (2.1.0)\n",
      "Requirement already satisfied: pygments>=2.15.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from devtools<0.13.0,>=0.12.2->graphrag) (2.15.1)\n",
      "Requirement already satisfied: marshmallow>=3.13.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from environs<12.0.0,>=11.0.0->graphrag) (3.23.1)\n",
      "Requirement already satisfied: POT<0.10,>=0.9 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (0.9.5)\n",
      "Requirement already satisfied: anytree<3.0.0,>=2.12.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (2.12.1)\n",
      "Requirement already satisfied: beartype<0.19.0,>=0.18.5 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (0.18.5)\n",
      "Requirement already satisfied: gensim<5.0.0,>=4.3.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (4.3.3)\n",
      "Requirement already satisfied: graspologic-native<2.0.0,>=1.2.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (1.2.1)\n",
      "Requirement already satisfied: hyppo<0.5.0,>=0.4.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (0.4.0)\n",
      "Requirement already satisfied: scikit-learn<2.0.0,>=1.4.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (1.5.2)\n",
      "Requirement already satisfied: scipy==1.12.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (1.12.0)\n",
      "Requirement already satisfied: seaborn<0.14.0,>=0.13.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (0.13.2)\n",
      "Requirement already satisfied: statsmodels<0.15.0,>=0.14.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from graspologic<4.0.0,>=3.4.1->graphrag) (0.14.4)\n",
      "Requirement already satisfied: deprecation in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (2.1.0)\n",
      "Requirement already satisfied: pylance==0.17.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (0.17.0)\n",
      "Requirement already satisfied: requests>=2.31.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (2.32.3)\n",
      "Requirement already satisfied: retry>=0.9.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (0.9.2)\n",
      "Requirement already satisfied: attrs>=21.3.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (24.2.0)\n",
      "Requirement already satisfied: packaging in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (24.1)\n",
      "Requirement already satisfied: cachetools in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (5.5.0)\n",
      "Requirement already satisfied: overrides>=0.7 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from lancedb<0.14.0,>=0.13.0->graphrag) (7.4.0)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (1.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (4.55.0)\n",
      "Requirement already satisfied: kiwisolver>=1.3.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (1.4.7)\n",
      "Requirement already satisfied: pillow>=8 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (11.0.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (3.2.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.0->graphrag) (2.9.0.post0)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from openai<2.0.0,>=1.51.2->graphrag) (4.6.2)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from openai<2.0.0,>=1.51.2->graphrag) (1.9.0)\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from openai<2.0.0,>=1.51.2->graphrag) (0.27.0)\n",
      "Requirement already satisfied: jiter<1,>=0.4.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from openai<2.0.0,>=1.51.2->graphrag) (0.7.1)\n",
      "Requirement already satisfied: sniffio in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from openai<2.0.0,>=1.51.2->graphrag) (1.3.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from pandas<3.0.0,>=2.2.3->graphrag) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from pandas<3.0.0,>=2.2.3->graphrag) (2024.2)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.9.2->graphrag) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.27.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.9.2->graphrag) (2.27.1)\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from rich<14.0.0,>=13.6.0->graphrag) (3.0.0)\n",
      "Requirement already satisfied: shellingham>=1.3.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from typer<0.13.0,>=0.12.5->graphrag) (1.5.4)\n",
      "Requirement already satisfied: numba>=0.51.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from umap-learn<0.6.0,>=0.5.6->graphrag) (0.60.0)\n",
      "Requirement already satisfied: pynndescent>=0.5 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from umap-learn<0.6.0,>=0.5.6->graphrag) (0.5.13)\n",
      "Requirement already satisfied: idna>=2.8 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from anyio<5,>=3.5.0->openai<2.0.0,>=1.51.2->graphrag) (3.7)\n",
      "Requirement already satisfied: six in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from anytree<3.0.0,>=2.12.1->graspologic<4.0.0,>=3.4.1->graphrag) (1.16.0)\n",
      "Requirement already satisfied: cffi>=1.12 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from cryptography>=2.5->azure-identity<2.0.0,>=1.17.1->graphrag) (1.17.1)\n",
      "Requirement already satisfied: smart-open>=1.8.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from gensim<5.0.0,>=4.3.2->graspologic<4.0.0,>=3.4.1->graphrag) (7.0.5)\n",
      "Requirement already satisfied: certifi in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.51.2->graphrag) (2024.8.30)\n",
      "Requirement already satisfied: httpcore==1.* in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.51.2->graphrag) (1.0.2)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.51.2->graphrag) (0.14.0)\n",
      "Requirement already satisfied: autograd>=1.3 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from hyppo<0.5.0,>=0.4.0->graspologic<4.0.0,>=3.4.1->graphrag) (1.7.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag) (2023.7.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag) (0.30.2)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.21.1->datashaper<0.0.50,>=0.0.49->graphrag) (0.10.6)\n",
      "Requirement already satisfied: mdurl~=0.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.6.0->graphrag) (0.1.2)\n",
      "Requirement already satisfied: PyJWT<3,>=1.0.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from PyJWT[crypto]<3,>=1.0.0->msal>=1.30.0->azure-identity<2.0.0,>=1.17.1->graphrag) (2.10.0)\n",
      "Requirement already satisfied: portalocker<3,>=1.4 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from msal-extensions>=1.2.0->azure-identity<2.0.0,>=1.17.1->graphrag) (2.10.1)\n",
      "Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from numba>=0.51.2->umap-learn<0.6.0,>=0.5.6->graphrag) (0.43.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from requests>=2.31.0->lancedb<0.14.0,>=0.13.0->graphrag) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from requests>=2.31.0->lancedb<0.14.0,>=0.13.0->graphrag) (2.2.3)\n",
      "Requirement already satisfied: decorator>=3.4.2 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from retry>=0.9.2->lancedb<0.14.0,>=0.13.0->graphrag) (5.1.1)\n",
      "Requirement already satisfied: py<2.0.0,>=1.4.26 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from retry>=0.9.2->lancedb<0.14.0,>=0.13.0->graphrag) (1.11.0)\n",
      "Requirement already satisfied: threadpoolctl>=3.1.0 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from scikit-learn<2.0.0,>=1.4.2->graspologic<4.0.0,>=3.4.1->graphrag) (3.5.0)\n",
      "Requirement already satisfied: patsy>=0.5.6 in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from statsmodels<0.15.0,>=0.14.2->graspologic<4.0.0,>=3.4.1->graphrag) (1.0.1)\n",
      "Requirement already satisfied: pycparser in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=2.5->azure-identity<2.0.0,>=1.17.1->graphrag) (2.21)\n",
      "Requirement already satisfied: wrapt in /root/miniconda3/envs/graphrag/lib/python3.11/site-packages (from smart-open>=1.8.1->gensim<5.0.0,>=4.3.2->graspologic<4.0.0,>=3.4.1->graphrag) (1.17.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install graphrag"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5404f422-18c6-4622-b887-40f0e792b6f3",
   "metadata": {},
   "source": [
    "本次公开课采用GraphRAG最新版 v1.1.0进行教学。新版特性如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b89017-6bf4-49ce-97b4-954ac7d5f924",
   "metadata": {},
   "source": [
    "- **简化项目设置**：引入了 `init` 命令，生成简化的 `settings.yml` 文件，预设核心配置，降低初始配置的复杂度。\n",
    "\n",
    "- **扩展命令行界面 (CLI)**：采用 Typer 框架，提供更丰富的内联文档和功能，使 CLI 的启动时间从平均 148 秒缩短至 2 秒。\n",
    "\n",
    "- **统一 API 层**：推出独立的 API 层，简化开发者集成，CLI 和加速器均基于此 API 构建，提供了与 API 交互的示例。\n",
    "\n",
    "- **简化数据模型**：优化数据结构，移除冗余字段，减少存储空间，提高读取速度和内存使用效率，输出的 Parquet 文件大小减少了 80%，总磁盘空间减少了 43%。\n",
    "\n",
    "- **优化向量存储**：默认在索引期间创建向量存储，避免后续处理，减少读取时间和内存占用，支持 LanceDB 和 Azure AI Search，默认使用 LanceDB 并写入本地数据库。\n",
    "\n",
    "- **扁平化代码结构**：简化代码库，减少层级深度，提升可维护性和可读性，合并工作流，减少未使用的输出，提高性能。\n",
    "\n",
    "- **增量摄取**：新增 `update` 命令，智能合并新内容与现有索引，减少重新索引的需求，利用 LLM 缓存机制，降低成本，加快处理速度。\n",
    "\n",
    "这些改进旨在提升 GraphRAG 的易用性、性能和可扩展性，为开发者和用户提供更高效的工具。  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e8a1737-eb78-4567-baf1-346984fd63e0",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108190644608.png\" alt=\"image-20250108190644608\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217984bf-a96b-4658-b6f2-ee8adba1abb2",
   "metadata": {},
   "source": [
    "- **Step 2.创建检索项目文件夹**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b37e7a1-307d-4d7a-9abe-30fa332b849b",
   "metadata": {},
   "source": [
    "```bash\n",
    "mkdir -p ./openl/input\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29ba1cb8-bbd7-49a1-870f-7471f757ba63",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128155735674.png\" alt=\"image-20241128155735674\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce624d82-621f-41bc-ad44-6d8fd7e6a773",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128155813677.png\" alt=\"image-20241128155813677\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef06e085-fb8c-4c92-8cfb-70d9f91ea7d4",
   "metadata": {},
   "source": [
    "- **Step 3.上传数据集**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ab0e0a-c681-44bc-bda1-a4c90af9b868",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128161523454.png\" alt=\"image-20241128161523454\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e79f87-77c0-4816-a223-808eefb74168",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128161608651.png\" alt=\"image-20241128161608651\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fbe9242-e645-496c-80f6-beb2fda26655",
   "metadata": {},
   "source": [
    "> 相关数据集已同步至本节公开课的课件包，扫码添加老师助理即可领取：\n",
    "> <center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/c6847a817fd7efd0cddcb1bcac217c3.png\" alt=\"c6847a817fd7efd0cddcb1bcac217c3\" style=\"zoom:55%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173c303e-dcb9-4957-a38c-1d128bb50ed0",
   "metadata": {},
   "source": [
    "- **Step 4.初始化项目文件**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483b8d63-aca8-401b-ae0c-956ca0674d20",
   "metadata": {},
   "source": [
    "```bash\n",
    "graphrag init --root ./openl\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "74f7323d-044c-4a2f-84a8-2b916d42fd6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2KInitializing project at \u001b[35m/root/autodl-tmp/graphrag/\u001b[0m\u001b[95mopenl\u001b[0m\n",
      "⠋ GraphRAG Indexer "
     ]
    }
   ],
   "source": [
    "!graphrag init --root ./openl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18e0c2ce-4298-49f5-821f-c10da77e7d69",
   "metadata": {},
   "source": [
    "- **Step 5.修改项目配置**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15014044-85f6-4698-a9da-8877cc536931",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128160243432.png\" alt=\"image-20241128160243432\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406082a5-17e2-45e4-a93a-954f3311deeb",
   "metadata": {},
   "source": [
    "打开.env文件，填写DeepSeek API-KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3c7a84-22dc-4add-9a19-f62c81c19f40",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108183829994.png\" alt=\"image-20250108183829994\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206fd362-7b56-4f7b-aa6a-64836457558f",
   "metadata": {},
   "source": [
    "打开setting.yaml文件，填写模型名称和反向代理地址："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a5e0c40-071f-495a-9d3a-bb33530d1375",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108184104855.png\" alt=\"image-20250108184104855\" style=\"zoom:50%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fdaacff-8bd3-4d55-8411-f22255234500",
   "metadata": {},
   "source": [
    "其中反向代理地址为`api_base: https://ai.devtool.tech/proxy/v1`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f991feb-723d-43b7-89a8-1c68f25bf9df",
   "metadata": {},
   "source": [
    "- **【可选】Step 6.验证API-KEY和反向代理地址是否可以正常运行**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e65470fc-a9dc-44c6-b7f4-bdc8545ca79d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ad93ee13-4d28-4e3c-9914-a875e00cf603",
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = 'your-openai-api-key'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "57dacd47-ab61-497d-abd6-19d91880c3d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 实例化客户端\n",
    "client = OpenAI(api_key=api_key, \n",
    "                base_url=\"https://ai.devtool.tech/proxy/v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ad81fa60-b8d6-4abd-a4be-b0f138c59470",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 调用 GPT-4o-mini 模型\n",
    "response = client.chat.completions.create(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"你好，好久不见!\"}\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7ea09c3e-eb88-4455-92a7-cdcab249e73a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好！好久不见！你最近怎么样？有什么新鲜事分享吗？\n"
     ]
    }
   ],
   "source": [
    "# 输出生成的响应内容\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "381250ef-bb65-4421-bee7-0925c24c989a",
   "metadata": {},
   "source": [
    "然后查看当前API-KEY可以调用的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b4b466a3-1861-430d-b60c-c9c0d5d2a9d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "models_list = client.models.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3291e9de-d0b9-4aca-8114-8a9caf9f0e05",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Model(id='gpt-4o-2024-08-06', created=1722814719, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o', created=1715367049, object='model', owned_by='system'),\n",
       " Model(id='o1-mini-2024-09-12', created=1725648979, object='model', owned_by='system'),\n",
       " Model(id='dall-e-2', created=1698798177, object='model', owned_by='system'),\n",
       " Model(id='gpt-4-1106-preview', created=1698957206, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo', created=1677610602, object='model', owned_by='openai'),\n",
       " Model(id='text-embedding-3-large', created=1705953180, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo-0125', created=1706048358, object='model', owned_by='system'),\n",
       " Model(id='babbage-002', created=1692634615, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-2024-11-20', created=1731975040, object='model', owned_by='system'),\n",
       " Model(id='gpt-4-turbo-preview', created=1706037777, object='model', owned_by='system'),\n",
       " Model(id='davinci-002', created=1692634301, object='model', owned_by='system'),\n",
       " Model(id='gpt-4-0125-preview', created=1706037612, object='model', owned_by='system'),\n",
       " Model(id='whisper-1', created=1677532384, object='model', owned_by='openai-internal'),\n",
       " Model(id='gpt-4o-mini', created=1721172741, object='model', owned_by='system'),\n",
       " Model(id='dall-e-3', created=1698785189, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-mini-2024-07-18', created=1721172717, object='model', owned_by='system'),\n",
       " Model(id='o1-preview', created=1725648897, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo-16k', created=1683758102, object='model', owned_by='openai-internal'),\n",
       " Model(id='o1-preview-2024-09-12', created=1725648865, object='model', owned_by='system'),\n",
       " Model(id='o1-mini', created=1725649008, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-2024-05-13', created=1715368132, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-realtime-preview', created=1727659998, object='model', owned_by='system'),\n",
       " Model(id='tts-1-hd-1106', created=1699053533, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-realtime-preview-2024-10-01', created=1727131766, object='model', owned_by='system'),\n",
       " Model(id='gpt-4', created=1687882411, object='model', owned_by='openai'),\n",
       " Model(id='gpt-4-0613', created=1686588896, object='model', owned_by='openai'),\n",
       " Model(id='text-embedding-ada-002', created=1671217299, object='model', owned_by='openai-internal'),\n",
       " Model(id='text-embedding-3-small', created=1705948997, object='model', owned_by='system'),\n",
       " Model(id='tts-1-hd', created=1699046015, object='model', owned_by='system'),\n",
       " Model(id='gpt-4-turbo', created=1712361441, object='model', owned_by='system'),\n",
       " Model(id='gpt-4-turbo-2024-04-09', created=1712601677, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo-1106', created=1698959748, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo-instruct', created=1692901427, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-audio-preview', created=1727460443, object='model', owned_by='system'),\n",
       " Model(id='gpt-4o-audio-preview-2024-10-01', created=1727389042, object='model', owned_by='system'),\n",
       " Model(id='tts-1', created=1681940951, object='model', owned_by='openai-internal'),\n",
       " Model(id='tts-1-1106', created=1699053241, object='model', owned_by='system'),\n",
       " Model(id='gpt-3.5-turbo-instruct-0914', created=1694122472, object='model', owned_by='system'),\n",
       " Model(id='chatgpt-4o-latest', created=1723515131, object='model', owned_by='system')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models_list.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b03feaa-6c8f-4311-ac83-48be04447fe4",
   "metadata": {},
   "source": [
    "#### 2.GraphRAG索引Indexing过程执行"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d470bf60-6812-4696-b108-b756105e6ff0",
   "metadata": {},
   "source": [
    "&emsp;&emsp;一切准备就绪后，即可开始执行GraphRAG索引过程。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3285e7ca-8d4e-4b61-bf1b-4549f15cfcb3",
   "metadata": {},
   "source": [
    "- **Step 7.借助GraphRAG脚本自动执行indexing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7baea274-1d70-4c31-9526-f7229b19953e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[2KLogging enabled at \u001b[35m/root/autodl-tmp/graphrag/openl/logs/\u001b[0m\u001b[95mindexing-engine.log\u001b[0m\n",
      "\u001b[2K⠙ GraphRAG Indexer \n",
      "\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer e.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_base_text_units\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KEmpty DataFrame\n",
      "Columns: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "Index: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_documents\u001b[0m\n",
      "⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                 id  \u001b[33m...\u001b[0m                                      \n",
      "text_unit_ids\n",
      "\u001b[1;36m0\u001b[0m  c546fa97b72c8b72b7efb0c1ac45cb1d  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3, \n",
      "b1fce21a45f\u001b[33m...\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m1\u001b[0m rows x \u001b[1;36m5\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer ━\u001b[0m \u001b[35m  0%\u001b[0m \u001b[36m-:--:--\u001b[0m \u001b[33m0:00:00\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠹ GraphRAG Indexer ━\u001b[0m \u001b[35m  0%\u001b[0m \u001b[36m-:--:--\u001b[0m \u001b[33m0:00:01\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer ━\u001b[0m \u001b[35m  0%\u001b[0m \u001b[36m-:--:--\u001b[0m \u001b[33m0:00:02\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer [90m━━━━━━━━━━━━━━━\u001b[0m \u001b[35m 25%\u001b[0m \u001b[36m-:--:--\u001b[0m \u001b[33m0:00:03\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[35m 50%\u001b[0m \u001b[36m0:00:01\u001b[0m \u001b[33m0:00:03\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[35m 50%\u001b[0m \u001b[36m0:00:01\u001b[0m \u001b[33m0:00:03\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[35m 50%\u001b[0m \u001b[36m0:00:01\u001b[0m \u001b[33m0:00:04\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[35m 50%\u001b[0m \u001b[36m0:00:01\u001b[0m \u001b[33m0:00:05\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[35m 50%\u001b[0m \u001b[36m0:00:01\u001b[0m \u001b[33m0:00:06\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer [91m╸\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[35m 75%\u001b[0m \u001b[36m0:00:03\u001b[0m \u001b[33m0:00:07\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer [91m╸\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[35m 75%\u001b[0m \u001b[36m0:00:03\u001b[0m \u001b[33m0:00:07\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer [91m╸\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[35m 75%\u001b[0m \u001b[36m0:00:03\u001b[0m \u001b[33m0:00:08\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer [91m╸\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[35m 75%\u001b[0m \u001b[36m0:00:03\u001b[0m \u001b[33m0:00:09\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "└── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer [91m╸\u001b[0m\u001b[90m━━━━━\u001b[0m \u001b[35m 75%\u001b[0m \u001b[36m0:00:03\u001b[0m \u001b[33m0:00:10\u001b[0m\n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_base_entity_graph\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KEmpty DataFrame\n",
      "Columns: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "Index: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_entities\u001b[0m\n",
      "⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                  id  \u001b[33m...\u001b[0m                                      \n",
      "text_unit_ids\n",
      "\u001b[1;36m0\u001b[0m   22113170e6c742ba940314e15c401dc1  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3, \n",
      "08530861b09\u001b[33m...\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m   23e8813975cc43d7921375da3c781c3c  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3, \n",
      "08530861b09\u001b[33m...\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m   3c4aa4fadb5c4b3f9244a7bb00598cab  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m3\u001b[0m   0ebac52d90264cef8bf563ffe894bbe5  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m4\u001b[0m   d45fe94d4f2840b48ebd2564ddc85e56  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m5\u001b[0m   8979a4d40b04416a91f084e057be0c98  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m6\u001b[0m   5a57bda8d0534c43a2591fde9519ef2e  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m7\u001b[0m   9ac71842fd5b47628a2cc43bb13b1290  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m8\u001b[0m   71c5eabc1ce9490f85f9357f6b7f309b  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m9\u001b[0m   de19324beea8438497852259c73c452a  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m10\u001b[0m  67f0629e10054ba49b2d1bfd08985927  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m11\u001b[0m rows x \u001b[1;36m6\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_nodes\u001b[0m\n",
      "⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                  id  human_readable_id  \u001b[33m...\u001b[0m  x  y\n",
      "\u001b[1;36m0\u001b[0m   22113170e6c742ba940314e15c401dc1                  \u001b[1;36m0\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m   23e8813975cc43d7921375da3c781c3c                  \u001b[1;36m1\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m   3c4aa4fadb5c4b3f9244a7bb00598cab                  \u001b[1;36m2\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m3\u001b[0m   0ebac52d90264cef8bf563ffe894bbe5                  \u001b[1;36m3\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m4\u001b[0m   d45fe94d4f2840b48ebd2564ddc85e56                  \u001b[1;36m4\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m5\u001b[0m   8979a4d40b04416a91f084e057be0c98                  \u001b[1;36m5\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m6\u001b[0m   5a57bda8d0534c43a2591fde9519ef2e                  \u001b[1;36m6\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m7\u001b[0m   9ac71842fd5b47628a2cc43bb13b1290                  \u001b[1;36m7\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m8\u001b[0m   71c5eabc1ce9490f85f9357f6b7f309b                  \u001b[1;36m8\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m9\u001b[0m   de19324beea8438497852259c73c452a                  \u001b[1;36m9\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m10\u001b[0m  67f0629e10054ba49b2d1bfd08985927                 \u001b[1;36m10\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m11\u001b[0m  22113170e6c742ba940314e15c401dc1                  \u001b[1;36m0\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m12\u001b[0m  23e8813975cc43d7921375da3c781c3c                  \u001b[1;36m1\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m13\u001b[0m  3c4aa4fadb5c4b3f9244a7bb00598cab                  \u001b[1;36m2\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m14\u001b[0m  0ebac52d90264cef8bf563ffe894bbe5                  \u001b[1;36m3\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m15\u001b[0m  d45fe94d4f2840b48ebd2564ddc85e56                  \u001b[1;36m4\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m16\u001b[0m  8979a4d40b04416a91f084e057be0c98                  \u001b[1;36m5\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m17\u001b[0m  5a57bda8d0534c43a2591fde9519ef2e                  \u001b[1;36m6\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m18\u001b[0m  9ac71842fd5b47628a2cc43bb13b1290                  \u001b[1;36m7\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m19\u001b[0m  71c5eabc1ce9490f85f9357f6b7f309b                  \u001b[1;36m8\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m20\u001b[0m  de19324beea8438497852259c73c452a                  \u001b[1;36m9\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\u001b[1;36m21\u001b[0m  67f0629e10054ba49b2d1bfd08985927                 \u001b[1;36m10\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m0\u001b[0m  \u001b[1;36m0\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m22\u001b[0m rows x \u001b[1;36m8\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_communities\u001b[0m\n",
      "⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                     id  human_readable_id  \u001b[33m...\u001b[0m      period  \n",
      "size\n",
      "\u001b[1;36m0\u001b[0m  \u001b[93m17bfead0-7ccc-438a-922b-5cd6deea39ea\u001b[0m                  \u001b[1;36m0\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m    \n",
      "\u001b[1;36m11\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m  \u001b[93m8eb758a5-95a9-4e42-93e2-5ac1adff8a42\u001b[0m                  \u001b[1;36m1\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m     \n",
      "\u001b[1;36m7\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m  \u001b[93m7b75cf95-4504-47a7-8b11-7c90fe66109b\u001b[0m                  \u001b[1;36m2\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m     \n",
      "\u001b[1;36m4\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m3\u001b[0m rows x \u001b[1;36m10\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠹ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_relationships\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                 id  \u001b[33m...\u001b[0m                                      \n",
      "text_unit_ids\n",
      "\u001b[1;36m0\u001b[0m  813887b9c3f54445832a00a8d1b47264  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3, \n",
      "08530861b09\u001b[33m...\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m  16f369775666497b8ece38456683313e  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m  6e17fd4dce3a432195f428ec5287c71d  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m3\u001b[0m  121cc7bef78848b69ec89e082cb86a54  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m4\u001b[0m  294c8d020d0b4b389070c7c1959f1cc7  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m5\u001b[0m  6110c0c995e34a3896bbc4a0274ecc6f  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m6\u001b[0m  188cab571d5c4b0197172dd2deab34a5  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m0653a697b64dd8c029503ffc22af9ec3\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m7\u001b[0m  4979732df79645ed88eea6e957cab2c7  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m8\u001b[0m  8a02e132514440d99006833312db5a5c  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\u001b[1;36m9\u001b[0m  3584cb0ca80a463c8e658b17bc7ef254  \u001b[33m...\u001b[0m                 \n",
      "\u001b[1m[\u001b[0m08530861b09098077a0ebcd53ab3ae62\u001b[1m]\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m10\u001b[0m rows x \u001b[1;36m8\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_text_units\u001b[0m\n",
      "⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                 id  \u001b[33m...\u001b[0m                                   \n",
      "relationship_ids\n",
      "\u001b[1;36m0\u001b[0m  0653a697b64dd8c029503ffc22af9ec3  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m813887b9c3f54445832a00a8d1b47264, \n",
      "16f36977566\u001b[33m...\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m  b1fce21a45f01c5ef14bafc9fe3bab1d  \u001b[33m...\u001b[0m                                        \n",
      "\u001b[3;35mNone\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m  08530861b09098077a0ebcd53ab3ae62  \u001b[33m...\u001b[0m  \u001b[1m[\u001b[0m813887b9c3f54445832a00a8d1b47264, \n",
      "4979732df79\u001b[33m...\u001b[0m\n",
      "\u001b[1;36m3\u001b[0m  e97cc7c63047f6471b4beac2f375eae0  \u001b[33m...\u001b[0m                                        \n",
      "\u001b[3;35mNone\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m4\u001b[0m rows x \u001b[1;36m7\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠹ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠧ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠹ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mcreate_final_community_reports\u001b[0m\n",
      "⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K                                     id  human_readable_id  \u001b[33m...\u001b[0m      period  \n",
      "size\n",
      "\u001b[1;36m0\u001b[0m  \u001b[93m100a5af4-6579-4fdc-afba-78565d6fe062\u001b[0m                  \u001b[1;36m1\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m     \n",
      "\u001b[1;36m7\u001b[0m\n",
      "\u001b[1;36m1\u001b[0m  \u001b[93m0e54e2ef-44ef-4bfa-8a8a-fd06f79ed338\u001b[0m                  \u001b[1;36m2\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m     \n",
      "\u001b[1;36m4\u001b[0m\n",
      "\u001b[1;36m2\u001b[0m  \u001b[93me527793c-b726-41be-892f-979334ae5073\u001b[0m                  \u001b[1;36m0\u001b[0m  \u001b[33m...\u001b[0m  \u001b[1;36m2024\u001b[0m-\u001b[1;36m11\u001b[0m-\u001b[1;36m28\u001b[0m    \n",
      "\u001b[1;36m11\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[1;36m3\u001b[0m rows x \u001b[1;36m13\u001b[0m columns\u001b[1m]\u001b[0m\n",
      "⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠴ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠇ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠙ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\u001b[0m\u001b[38;5;8m[\u001b[0m2024-11-28T08:20:05Z \u001b[0m\u001b[33mWARN \u001b[0m lance::dataset\u001b[0m\u001b[38;5;8m]\u001b[0m No existing dataset at /root/autodl-tmp/graphrag/openl/output/lancedb/default-text_unit-text.lance, it will be created\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠸ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠦ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠼ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\u001b[0m\u001b[38;5;8m[\u001b[0m2024-11-28T08:20:08Z \u001b[0m\u001b[33mWARN \u001b[0m lance::dataset\u001b[0m\u001b[38;5;8m]\u001b[0m No existing dataset at /root/autodl-tmp/graphrag/openl/output/lancedb/default-entity-description.lance, it will be created\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠹ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠏ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\u001b[0m\u001b[38;5;8m[\u001b[0m2024-11-28T08:20:09Z \u001b[0m\u001b[33mWARN \u001b[0m lance::dataset\u001b[0m\u001b[38;5;8m]\u001b[0m No existing dataset at /root/autodl-tmp/graphrag/openl/output/lancedb/default-community-full_content.lance, it will be created\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K🚀 \u001b[32mgenerate_text_embeddings\u001b[0m\n",
      "⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KEmpty DataFrame\n",
      "Columns: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "Index: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
      "⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K⠋ GraphRAG Indexer \n",
      "├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) \u001b[90m━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m…\u001b[0m \u001b[33m0…\u001b[0m\n",
      "├── create_base_text_units\n",
      "├── create_final_documents\n",
      "├── create_base_entity_graph\n",
      "├── create_final_entities\n",
      "├── create_final_nodes\n",
      "├── create_final_communities\n",
      "├── create_final_relationships\n",
      "├── create_final_text_units\n",
      "├── create_final_community_reports\n",
      "└── generate_text_embeddings\n",
      "\u001b[?25h🚀 \u001b[32mAll workflows completed successfully.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!graphrag index --root ./openl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "584fe51a-95cf-4390-b534-b8b9c238db37",
   "metadata": {},
   "source": [
    "该命令也可以在终端内运行，结果如图："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aa9e3c0-db00-4027-864b-a2e48dc5b3de",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108152705197.png\" alt=\"image-20250108152705197\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c5817a5-9023-4370-ab34-2b506b1876f2",
   "metadata": {},
   "source": [
    "运行结束后，各知识图谱相关数据集都保存在output文件夹中："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a09d086-7339-450b-87cc-e3caa2ac0525",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20241128162059537.png\" alt=\"image-20241128162059537\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cab18325-3f3a-4e39-8ad2-11bff2c1223b",
   "metadata": {},
   "source": [
    "<center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108184159353.png\" alt=\"image-20250108184159353\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5dbf286-2646-46d2-b6af-7aea0690157c",
   "metadata": {},
   "source": [
    "&emsp;&emsp;**GraphRAG在索引阶段构建的知识图谱**的确是以 **Parquet** 格式保存的！索引阶段的输出文件中，Parquet 文件存储了知识图谱的各个核心组成部分，例如**实体**、**关系**、**社区信息**以及**文本单元**等。这些文件共同组成了一个完整的知识图谱。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "946abdd3-7be6-4fdd-b5ff-9626c4c5f871",
   "metadata": {},
   "source": [
    "索引阶段的主要输出内容如下：\n",
    "\n",
    "1. **实体表（Nodes Table）**\n",
    "   - **文件名**：`create_final_nodes.parquet`\n",
    "   - **内容**：知识图谱中的实体节点（例如：人、地点、组织）。\n",
    "   - **包含信息**：\n",
    "     - 实体的名称（如 \"John Doe\"）。\n",
    "     - 实体的类别（如 \"PERSON\", \"ORGANIZATION\", \"LOCATION\"）。\n",
    "     - 与社区相关的信息（如实体所属的社区）。\n",
    "     - 实体的度数（degree），表示该实体在图谱中的连接数。\n",
    "\n",
    "2. **关系表（Relationships Table）**\n",
    "   - **文件名**：`create_final_relationships.parquet`\n",
    "   - **内容**：知识图谱中实体之间的关系（即图谱的边）。\n",
    "   - **包含信息**：\n",
    "     - 两个实体之间的关系描述（例如 \"works for\", \"lives in\"）。\n",
    "     - 关系的强度（数值化，用于衡量关系的显著性或重要性）。\n",
    "\n",
    "3. **嵌入向量表（Entity Embedding Table）**\n",
    "   - **文件名**：`create_final_entities.parquet`\n",
    "   - **内容**：实体的语义嵌入，用于表示实体的语义信息。\n",
    "   - **用途**：支持语义搜索（通过嵌入计算实体之间的相似性）。\n",
    "\n",
    "4. **社区报告表（Community Reports Table）**\n",
    "   - **文件名**：`create_final_community_reports.parquet`\n",
    "   - **内容**：社区的摘要信息。\n",
    "   - **用途**：支持全局搜索（通过社区信息回答关于数据集整体的问题）。\n",
    "\n",
    "5. **文本单元表（Text Units Table）**\n",
    "   - **文件名**：`create_final_text_units.parquet`\n",
    "   - **内容**：被切分的原始文本单元（TextUnits）。\n",
    "   - **用途**：将知识图谱和原始文本结合，为 LLM 提供上下文支持。\n",
    "\n",
    "6. **社区表（Community Table）**\n",
    "   - **文件名**：`create_final_Communities.parquet`\n",
    "   - **内容**：每个社区基本情况。\n",
    "    \n",
    "7. **文件表（Documents Table）**\n",
    "   - **文件名**：`create_final_documents.parquet`\n",
    "   - **内容**：用于记录所有参与知识图谱构建的文件情况。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ffb01c-727e-4d7f-b128-3da6b51039a7",
   "metadata": {},
   "source": [
    ">  **为什么用 Parquet 格式保存知识图谱？**\n",
    ">\n",
    ">1. **高效存储**：\n",
    ">   - 知识图谱中的数据通常是结构化的，包含大量的实体、关系、嵌入等。\n",
    ">   - Parquet 的列式存储能够显著减少磁盘占用，同时提高读取效率。\n",
    ">\n",
    ">2. **快速读取**：\n",
    ">   - 查询阶段需要快速加载实体、关系、嵌入等数据到内存中。\n",
    ">   - Parquet 支持按需加载所需的列，避免了不必要的数据读取。\n",
    ">\n",
    ">3. **兼容性好**：\n",
    ">   - Parquet 是一个开放的标准，广泛支持各种数据处理工具（如 Pandas、Spark、Hadoop）。\n",
    ">   - GraphRAG 可以在 Python 中使用 Pandas 或其他工具轻松读取这些文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afb8e5ee-7e3f-49f9-b25c-055c4fb90e3c",
   "metadata": {},
   "source": [
    "#### 3.查看知识图谱相关表格"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8978930a-d674-4c5d-af58-bbb80dfd026f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f5073fd-7523-4825-a527-d907fc323af7",
   "metadata": {},
   "source": [
    "- 文件表（Documents Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "63569650-1c41-448b-b68e-fa757bc2a024",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>text_unit_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311cc...</td>\n",
       "      <td>1</td>\n",
       "      <td>ID3、C4.5决策树的建模流程.txt</td>\n",
       "      <td>Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  id  human_readable_id  \\\n",
       "0  2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311cc...                  1   \n",
       "\n",
       "                  title                                               text  \\\n",
       "0  ID3、C4.5决策树的建模流程.txt  Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...   \n",
       "\n",
       "                                       text_unit_ids  \n",
       "0  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents_df = pd.read_parquet(\"./openl/output/create_final_documents.parquet\")\n",
    "documents_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6206e21d-4cbe-46d5-ac63-ab05670b1793",
   "metadata": {},
   "source": [
    "- 文本单元表（TEXT UNIT Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dcb5eaf0-079b-4365-bf44-eb1487076c94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>text</th>\n",
       "      <th>n_tokens</th>\n",
       "      <th>document_ids</th>\n",
       "      <th>entity_ids</th>\n",
       "      <th>relationship_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992ad...</td>\n",
       "      <td>1</td>\n",
       "      <td>Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7a8436d7970da4b0e5c837b0a3eaaa196e6e2284c7c754...</td>\n",
       "      <td>2</td>\n",
       "      <td>8961919\\n然后即可算出按照如此规则进行数据集划分，最终能够减少的不纯度数值：\\n# ...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1141affdc56bf8f5092b950b57febdedb884673ec6b23d...</td>\n",
       "      <td>3</td>\n",
       "      <td>4.5决策树的基本建模流程\\n作为ID3的改进版算法，C4.5在ID3的基础上进行了三个方面...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[651dfca9-7a44-4d11-aed2-c00ce76f936d, 3ad66ad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b5df95b6ec7e3aaea3f67db4914e1b2e874f3ac77fc3c8...</td>\n",
       "      <td>4</td>\n",
       "      <td>集划分。\\n\\nC4.5的连续变量处理方法\\nC4.5允许带入连续变量进行建模，并且围绕连续...</td>\n",
       "      <td>558</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[651dfca9-7a44-4d11-aed2-c00ce76f936d, 9ddcf83...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  id  human_readable_id  \\\n",
       "0  4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992ad...                  1   \n",
       "1  7a8436d7970da4b0e5c837b0a3eaaa196e6e2284c7c754...                  2   \n",
       "2  1141affdc56bf8f5092b950b57febdedb884673ec6b23d...                  3   \n",
       "3  b5df95b6ec7e3aaea3f67db4914e1b2e874f3ac77fc3c8...                  4   \n",
       "\n",
       "                                                text  n_tokens  \\\n",
       "0  Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...      1200   \n",
       "1  8961919\\n然后即可算出按照如此规则进行数据集划分，最终能够减少的不纯度数值：\\n# ...      1200   \n",
       "2  4.5决策树的基本建模流程\\n作为ID3的改进版算法，C4.5在ID3的基础上进行了三个方面...      1200   \n",
       "3  集划分。\\n\\nC4.5的连续变量处理方法\\nC4.5允许带入连续变量进行建模，并且围绕连续...       558   \n",
       "\n",
       "                                        document_ids  \\\n",
       "0  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "1  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "2  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "3  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "\n",
       "                                          entity_ids  \\\n",
       "0  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "1  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "2  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "3  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "\n",
       "                                    relationship_ids  \n",
       "0  [a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...  \n",
       "1  [a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...  \n",
       "2  [651dfca9-7a44-4d11-aed2-c00ce76f936d, 3ad66ad...  \n",
       "3  [651dfca9-7a44-4d11-aed2-c00ce76f936d, 9ddcf83...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_unit_df = pd.read_parquet(\"./openl/output/create_final_text_units.parquet\")\n",
    "text_unit_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74fc076a-581c-4f17-81f7-5f0396a65497",
   "metadata": {},
   "source": [
    "- 实体嵌入表（ENTITIES Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b5fa3c29-9cd9-4a94-be9d-f1c80c075916",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>title</th>\n",
       "      <th>type</th>\n",
       "      <th>description</th>\n",
       "      <th>text_unit_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35bbb43d-b9f4-4a4d-bd53-7059efc44056</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>ORGANIZATION</td>\n",
       "      <td>ID3 is a classic decision tree algorithm prima...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1535df1-c0e1-4bd4-9b18-49b05918a0aa</td>\n",
       "      <td>1</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>ORGANIZATION</td>\n",
       "      <td>C4.5 is an advanced decision tree algorithm an...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9afdfaaf-37d8-4990-9406-a62f37c2d64e</td>\n",
       "      <td>2</td>\n",
       "      <td>CART</td>\n",
       "      <td>ORGANIZATION</td>\n",
       "      <td>CART (Classification and Regression Trees) is ...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b3651a61-5626-4162-9f1e-432e6d0e9fe5</td>\n",
       "      <td>3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>ORGANIZATION</td>\n",
       "      <td>SKLEARN, also known as Scikit-learn, is a wide...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1</td>\n",
       "      <td>4</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>ORGANIZATION</td>\n",
       "      <td>NumPy is a Python library used for numerical c...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id    title  \\\n",
       "0  35bbb43d-b9f4-4a4d-bd53-7059efc44056                  0      ID3   \n",
       "1  f1535df1-c0e1-4bd4-9b18-49b05918a0aa                  1     C4.5   \n",
       "2  9afdfaaf-37d8-4990-9406-a62f37c2d64e                  2     CART   \n",
       "3  b3651a61-5626-4162-9f1e-432e6d0e9fe5                  3  SKLEARN   \n",
       "4  c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1                  4    NUMPY   \n",
       "\n",
       "           type                                        description  \\\n",
       "0  ORGANIZATION  ID3 is a classic decision tree algorithm prima...   \n",
       "1  ORGANIZATION  C4.5 is an advanced decision tree algorithm an...   \n",
       "2  ORGANIZATION  CART (Classification and Regression Trees) is ...   \n",
       "3  ORGANIZATION  SKLEARN, also known as Scikit-learn, is a wide...   \n",
       "4  ORGANIZATION  NumPy is a Python library used for numerical c...   \n",
       "\n",
       "                                       text_unit_ids  \n",
       "0  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "1  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "2  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "3  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "4  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_embedding_df = pd.read_parquet(\"./openl/output/create_final_entities.parquet\")\n",
    "entity_embedding_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bf52c64-c365-4b93-80a2-76c069bae8e5",
   "metadata": {},
   "source": [
    "- 实体表（Nodes Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fdc32b6b-1d9c-4773-b0f0-104e363cf24e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>title</th>\n",
       "      <th>community</th>\n",
       "      <th>level</th>\n",
       "      <th>degree</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35bbb43d-b9f4-4a4d-bd53-7059efc44056</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1535df1-c0e1-4bd4-9b18-49b05918a0aa</td>\n",
       "      <td>1</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9afdfaaf-37d8-4990-9406-a62f37c2d64e</td>\n",
       "      <td>2</td>\n",
       "      <td>CART</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b3651a61-5626-4162-9f1e-432e6d0e9fe5</td>\n",
       "      <td>3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1</td>\n",
       "      <td>4</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id    title  \\\n",
       "0  35bbb43d-b9f4-4a4d-bd53-7059efc44056                  0      ID3   \n",
       "1  f1535df1-c0e1-4bd4-9b18-49b05918a0aa                  1     C4.5   \n",
       "2  9afdfaaf-37d8-4990-9406-a62f37c2d64e                  2     CART   \n",
       "3  b3651a61-5626-4162-9f1e-432e6d0e9fe5                  3  SKLEARN   \n",
       "4  c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1                  4    NUMPY   \n",
       "\n",
       "   community  level  degree  x  y  \n",
       "0          0      0       9  0  0  \n",
       "1          1      0      11  0  0  \n",
       "2          0      0       8  0  0  \n",
       "3          0      0       3  0  0  \n",
       "4          2      0       4  0  0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_df = pd.read_parquet(\"./openl/output/create_final_nodes.parquet\")\n",
    "entity_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0d48cc10-80a1-4dcf-91f7-1c89679e0c12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>title</th>\n",
       "      <th>community</th>\n",
       "      <th>level</th>\n",
       "      <th>degree</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35bbb43d-b9f4-4a4d-bd53-7059efc44056</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1535df1-c0e1-4bd4-9b18-49b05918a0aa</td>\n",
       "      <td>1</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9afdfaaf-37d8-4990-9406-a62f37c2d64e</td>\n",
       "      <td>2</td>\n",
       "      <td>CART</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b3651a61-5626-4162-9f1e-432e6d0e9fe5</td>\n",
       "      <td>3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1</td>\n",
       "      <td>4</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>f7e0cfe0-7d50-4ea6-8bfe-4f252cf985cb</td>\n",
       "      <td>5</td>\n",
       "      <td>ML_BASIC_FUNCTION</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>a0684f1a-0a87-45dc-b240-55b3d04f1961</td>\n",
       "      <td>6</td>\n",
       "      <td>AGE</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>da7c9be6-2768-4baf-9185-5aa389fc91e7</td>\n",
       "      <td>7</td>\n",
       "      <td>INCOME</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>a048ca84-9e48-4beb-a8d7-2a5b3db76da5</td>\n",
       "      <td>8</td>\n",
       "      <td>STUDENT</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>590c9b35-7d6c-4e40-914c-2059472950c0</td>\n",
       "      <td>9</td>\n",
       "      <td>CREDIT_RATING</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ed5c4958-1ab3-4acd-889e-b9034ca07f35</td>\n",
       "      <td>10</td>\n",
       "      <td>GAIN RATIO</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>9d25b7b9-7d87-4f8b-ab6a-89adf3df59be</td>\n",
       "      <td>11</td>\n",
       "      <td>INFORMATION GAIN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>36a992dd-c7d3-4c93-8574-e6ce14233088</td>\n",
       "      <td>12</td>\n",
       "      <td>INFORMATION VALUE</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      id  human_readable_id  \\\n",
       "0   35bbb43d-b9f4-4a4d-bd53-7059efc44056                  0   \n",
       "1   f1535df1-c0e1-4bd4-9b18-49b05918a0aa                  1   \n",
       "2   9afdfaaf-37d8-4990-9406-a62f37c2d64e                  2   \n",
       "3   b3651a61-5626-4162-9f1e-432e6d0e9fe5                  3   \n",
       "4   c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1                  4   \n",
       "5   f7e0cfe0-7d50-4ea6-8bfe-4f252cf985cb                  5   \n",
       "6   a0684f1a-0a87-45dc-b240-55b3d04f1961                  6   \n",
       "7   da7c9be6-2768-4baf-9185-5aa389fc91e7                  7   \n",
       "8   a048ca84-9e48-4beb-a8d7-2a5b3db76da5                  8   \n",
       "9   590c9b35-7d6c-4e40-914c-2059472950c0                  9   \n",
       "10  ed5c4958-1ab3-4acd-889e-b9034ca07f35                 10   \n",
       "11  9d25b7b9-7d87-4f8b-ab6a-89adf3df59be                 11   \n",
       "12  36a992dd-c7d3-4c93-8574-e6ce14233088                 12   \n",
       "\n",
       "                title  community  level  degree  x  y  \n",
       "0                 ID3          0      0       9  0  0  \n",
       "1                C4.5          1      0      11  0  0  \n",
       "2                CART          0      0       8  0  0  \n",
       "3             SKLEARN          0      0       3  0  0  \n",
       "4               NUMPY          2      0       4  0  0  \n",
       "5   ML_BASIC_FUNCTION          2      0       1  0  0  \n",
       "6                 AGE          0      0       3  0  0  \n",
       "7              INCOME          0      0       3  0  0  \n",
       "8             STUDENT          0      0       3  0  0  \n",
       "9       CREDIT_RATING          0      0       3  0  0  \n",
       "10         GAIN RATIO          1      0       1  0  0  \n",
       "11   INFORMATION GAIN          1      0       2  0  0  \n",
       "12  INFORMATION VALUE          1      0       1  0  0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40e0d60-f6ec-48dd-b5a1-e14defe84fa9",
   "metadata": {},
   "source": [
    "关系表（Relationships Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ed8eeb1a-8d4a-4a7c-8429-626c972d7927",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>source</th>\n",
       "      <th>target</th>\n",
       "      <th>description</th>\n",
       "      <th>weight</th>\n",
       "      <th>combined_degree</th>\n",
       "      <th>text_unit_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>ID3 and C4.5 are both decision tree algorithms...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>20</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80278b16-6586-491e-af78-f73b39394df9</td>\n",
       "      <td>1</td>\n",
       "      <td>ID3</td>\n",
       "      <td>CART</td>\n",
       "      <td>ID3 and CART are both decision tree algorithms...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>17</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>409e56e1-d48a-4384-ba51-070fd10c58c1</td>\n",
       "      <td>2</td>\n",
       "      <td>ID3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>ID3 is not directly supported by Sklearn, but ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c54b57c8-c3bb-4aab-b8da-6a1e3ff95726</td>\n",
       "      <td>3</td>\n",
       "      <td>ID3</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>NumPy is used for entropy calculations in ID3 ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>13</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>651dfca9-7a44-4d11-aed2-c00ce76f936d</td>\n",
       "      <td>4</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>CART</td>\n",
       "      <td>C4.5 and CART are both decision tree algorithm...</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id source   target  \\\n",
       "0  a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543                  0    ID3     C4.5   \n",
       "1  80278b16-6586-491e-af78-f73b39394df9                  1    ID3     CART   \n",
       "2  409e56e1-d48a-4384-ba51-070fd10c58c1                  2    ID3  SKLEARN   \n",
       "3  c54b57c8-c3bb-4aab-b8da-6a1e3ff95726                  3    ID3    NUMPY   \n",
       "4  651dfca9-7a44-4d11-aed2-c00ce76f936d                  4   C4.5     CART   \n",
       "\n",
       "                                         description  weight  combined_degree  \\\n",
       "0  ID3 and C4.5 are both decision tree algorithms...    18.0               20   \n",
       "1  ID3 and CART are both decision tree algorithms...    14.0               17   \n",
       "2  ID3 is not directly supported by Sklearn, but ...     2.0               12   \n",
       "3  NumPy is used for entropy calculations in ID3 ...     2.0               13   \n",
       "4  C4.5 and CART are both decision tree algorithm...    17.0               19   \n",
       "\n",
       "                                       text_unit_ids  \n",
       "0  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "1  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "2  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "3  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "4  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relationships_df = pd.read_parquet(\"./openl/output/create_final_relationships.parquet\")\n",
    "relationships_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd109c6a-0e66-4c25-b66e-edd9d0961c1d",
   "metadata": {},
   "source": [
    "社区表（Community Table）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "95b8aaac-e313-4e03-9ee0-be8aa4cd64a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>community</th>\n",
       "      <th>parent</th>\n",
       "      <th>level</th>\n",
       "      <th>title</th>\n",
       "      <th>entity_ids</th>\n",
       "      <th>relationship_ids</th>\n",
       "      <th>text_unit_ids</th>\n",
       "      <th>period</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b73809f5-60cc-4a86-a52b-77c8954460ea</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Community 0</td>\n",
       "      <td>[a0684f1a-0a87-45dc-b240-55b3d04f1961, 9afdfaa...</td>\n",
       "      <td>[2858cef2-8514-4bb9-a8fc-aaf4249f2a62, 2e7eee3...</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d3467630-82c0-48da-9ef2-9d2aa65fd77f</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Community 1</td>\n",
       "      <td>[f1535df1-c0e1-4bd4-9b18-49b05918a0aa, ed5c495...</td>\n",
       "      <td>[6304470a-dfd0-4a9e-9232-dc6446864dd4, 7640403...</td>\n",
       "      <td>[1141affdc56bf8f5092b950b57febdedb884673ec6b23...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2750b356-bf1c-4985-bd66-240627d193fb</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Community 2</td>\n",
       "      <td>[c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1, f7e0cfe...</td>\n",
       "      <td>[6307647a-5e6d-4a12-9081-ee72b9b7e3fb]</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id  community  parent  \\\n",
       "0  b73809f5-60cc-4a86-a52b-77c8954460ea                  0          0      -1   \n",
       "1  d3467630-82c0-48da-9ef2-9d2aa65fd77f                  1          1      -1   \n",
       "2  2750b356-bf1c-4985-bd66-240627d193fb                  2          2      -1   \n",
       "\n",
       "   level        title                                         entity_ids  \\\n",
       "0      0  Community 0  [a0684f1a-0a87-45dc-b240-55b3d04f1961, 9afdfaa...   \n",
       "1      0  Community 1  [f1535df1-c0e1-4bd4-9b18-49b05918a0aa, ed5c495...   \n",
       "2      0  Community 2  [c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1, f7e0cfe...   \n",
       "\n",
       "                                    relationship_ids  \\\n",
       "0  [2858cef2-8514-4bb9-a8fc-aaf4249f2a62, 2e7eee3...   \n",
       "1  [6304470a-dfd0-4a9e-9232-dc6446864dd4, 7640403...   \n",
       "2             [6307647a-5e6d-4a12-9081-ee72b9b7e3fb]   \n",
       "\n",
       "                                       text_unit_ids      period  size  \n",
       "0  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  2025-01-08     7  \n",
       "1  [1141affdc56bf8f5092b950b57febdedb884673ec6b23...  2025-01-08     4  \n",
       "2  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  2025-01-08     2  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "community_df = pd.read_parquet(\"./openl/output/create_final_communities.parquet\")\n",
    "community_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0c0cb7b9-e34a-4456-952d-bb32200c18b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>community</th>\n",
       "      <th>parent</th>\n",
       "      <th>level</th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>full_content</th>\n",
       "      <th>rank</th>\n",
       "      <th>rank_explanation</th>\n",
       "      <th>findings</th>\n",
       "      <th>full_content_json</th>\n",
       "      <th>period</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c81a85511b7246ac8f93a15783e26855</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree Algorithms: ID3, C4.5, and CART</td>\n",
       "      <td>This community revolves around key decision tr...</td>\n",
       "      <td># Decision Tree Algorithms: ID3, C4.5, and CAR...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'ID3 is recognized as a class...</td>\n",
       "      <td>{\\n    \"title\": \"Decision Tree Algorithms: ID3...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0c4efb0094a64a3abd1a184b1baab37e</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>C4.5 and Decision Tree Algorithms</td>\n",
       "      <td>This community is centered around the C4.5 dec...</td>\n",
       "      <td># C4.5 and Decision Tree Algorithms\\n\\nThis co...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'C4.5 represents a significan...</td>\n",
       "      <td>{\\n    \"title\": \"C4.5 and Decision Tree Algori...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>983b4cf9540e4ee7820611ff6f6d33ba</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>NumPy and ML_Basic_Function Community</td>\n",
       "      <td>This community is centered around the NumPy li...</td>\n",
       "      <td># NumPy and ML_Basic_Function Community\\n\\nThi...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'NumPy is identified as a Pyt...</td>\n",
       "      <td>{\\n    \"title\": \"NumPy and ML_Basic_Function C...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 id  human_readable_id  community  parent  \\\n",
       "0  c81a85511b7246ac8f93a15783e26855                  0          0      -1   \n",
       "1  0c4efb0094a64a3abd1a184b1baab37e                  1          1      -1   \n",
       "2  983b4cf9540e4ee7820611ff6f6d33ba                  2          2      -1   \n",
       "\n",
       "   level                                          title  \\\n",
       "0      0  Decision Tree Algorithms: ID3, C4.5, and CART   \n",
       "1      0              C4.5 and Decision Tree Algorithms   \n",
       "2      0          NumPy and ML_Basic_Function Community   \n",
       "\n",
       "                                             summary  \\\n",
       "0  This community revolves around key decision tr...   \n",
       "1  This community is centered around the C4.5 dec...   \n",
       "2  This community is centered around the NumPy li...   \n",
       "\n",
       "                                        full_content  rank  \\\n",
       "0  # Decision Tree Algorithms: ID3, C4.5, and CAR...   7.5   \n",
       "1  # C4.5 and Decision Tree Algorithms\\n\\nThis co...   7.5   \n",
       "2  # NumPy and ML_Basic_Function Community\\n\\nThi...   7.5   \n",
       "\n",
       "                                    rank_explanation  \\\n",
       "0  The impact severity rating is high due to the ...   \n",
       "1  The impact severity rating is high due to the ...   \n",
       "2  The impact severity rating is high due to the ...   \n",
       "\n",
       "                                            findings  \\\n",
       "0  [{'explanation': 'ID3 is recognized as a class...   \n",
       "1  [{'explanation': 'C4.5 represents a significan...   \n",
       "2  [{'explanation': 'NumPy is identified as a Pyt...   \n",
       "\n",
       "                                   full_content_json      period  size  \n",
       "0  {\\n    \"title\": \"Decision Tree Algorithms: ID3...  2025-01-08     7  \n",
       "1  {\\n    \"title\": \"C4.5 and Decision Tree Algori...  2025-01-08     4  \n",
       "2  {\\n    \"title\": \"NumPy and ML_Basic_Function C...  2025-01-08     2  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "community_report_df = pd.read_parquet(\"./openl/output/create_final_community_reports.parquet\")\n",
    "community_report_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41361d7f-896c-4ef3-a5f0-699db9fe6077",
   "metadata": {},
   "source": [
    "是**社区报告（Community Reports）** 每一行代表一个社区的摘要信息，包含了与该社区相关的标题、摘要、实体、关系等详细内容。这些信息是 GraphRAG 在索引阶段基于知识图谱生成的高层次语义总结，用于帮助回答关于数据集整体的问题（例如全局搜索）。\n",
    "\n",
    " **表格字段的含义**\n",
    "\n",
    "1. **`id`**\n",
    "   - 每个社区的唯一标识符（UUID）。\n",
    "   - 这是系统内部生成的，通常用于追踪社区记录。\n",
    "\n",
    "2. **`human_readable_id`**\n",
    "   - 更容易理解的人类可读ID（数字化或短标识符）。\n",
    "   - 通常用于直观地区分社区。\n",
    "\n",
    "3. **`community`**\n",
    "   - 社区的编号，代表社区的分类或标识。\n",
    "\n",
    "4. **`level`**\n",
    "   - 社区层级（`COMMUNITY_LEVEL`）。\n",
    "   - 表示社区的聚类粒度。较低的值表示更抽象的层级，较高的值表示更具体的细分社区。\n",
    "\n",
    "5. **`title`**\n",
    "   - 社区的标题。\n",
    "   - 对该社区的简要描述，用于表示该社区的主题核心。例如：\n",
    "     - **`ID3 Decision Tree Algorithm Community`**\n",
    "     - **`C4.5 and Sklearn Integration`**\n",
    "\n",
    "6. **`summary`**\n",
    "   - 社区的摘要。\n",
    "   - 对该社区内容的进一步扩展描述，解释其主题或涵盖的主要内容。\n",
    "\n",
    "7. **`full_content`**\n",
    "   - 社区报告的完整内容。\n",
    "   - 包括更详细的信息，可能包含标题、摘要、重要发现等。\n",
    "\n",
    "8. **`rank`**\n",
    "   - 社区的排名或评分。\n",
    "   - 可能用来表示社区的重要性、影响力或相关性。\n",
    "\n",
    "9. **`rank_explanation`**\n",
    "   - 对排名的解释。\n",
    "   - 例如，“影响严重性从中等到高”。\n",
    "\n",
    "10. **`findings`**\n",
    "    - 社区的主要发现（JSON 格式）。\n",
    "    - 包含详细的解释或分析。例如：\n",
    "      - **ID3 算法是该社区的核心主题。**\n",
    "      - **C4.5 是 ID3 的改进版本，结合了 Sklearn 的功能。**\n",
    "\n",
    "11. **`full_content_json`**\n",
    "    - 社区完整内容的 JSON 表示。\n",
    "    - 如果需要以结构化方式处理社区内容，这个字段是关键。\n",
    "\n",
    "12. **`period`**\n",
    "    - 报告的时间戳或周期。\n",
    "    - 在你的示例中，是 `2024-11-26`，表示报告生成的日期。\n",
    "\n",
    "13. **`size`**\n",
    "    - 社区的大小。\n",
    "    - 表示该社区包含的实体或内容的数量。例如：\n",
    "      - **社区 0 包含 8 个内容。**\n",
    "      - **社区 1 包含 2 个内容。**\n",
    "\n",
    " **结果解释**\n",
    "从你的结果中可以看出：\n",
    "1. **社区 0**：\n",
    "   - 标题是“ID3 Decision Tree Algorithm Community”。\n",
    "   - 主题围绕 ID3 决策树算法。\n",
    "   - 内容提到了该算法的重要性，以及其对机器学习领域的影响。\n",
    "   - 社区大小为 8。\n",
    "\n",
    "2. **社区 1**：\n",
    "   - 标题是“C4.5 and Sklearn Integration”。\n",
    "   - 主题是 C4.5 算法和 Sklearn 的集成。\n",
    "   - 该社区比社区 0 更小（仅有 2 个内容）。\n",
    "   - 排名评分为 4.5，影响程度适中。\n",
    "\n",
    "**用途**\n",
    "社区报告在查询阶段可以用于：\n",
    "1. **全局搜索（Global Search）**：\n",
    "   - 回答关于数据集整体的问题，例如“这些文档的主要主题是什么？”\n",
    "2. **快速理解社区**：\n",
    "   - 帮助用户快速了解数据集中不同部分的主题和相关信息。\n",
    "3. **可视化与调试**：\n",
    "   - 将这些社区报告与知识图谱结合起来，可以直观地呈现社区的结构和语义关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f505429f-324b-4d51-a8af-6d4bbaac6c83",
   "metadata": {},
   "source": [
    "### 三、GraphRAG问答流程"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "68ab7201-7fd0-497d-ba89-e9d4841f77c4",
   "metadata": {},
   "source": [
    "> 除了使用Python代码环境进行问答外，若希望快速测试问答性能，也可以直接在命令行中，借助graphrag提供的脚本命令进行快速问答，例如：`graphrag query --root ./openl/input --method local --query \"请帮我介绍下ID3算法\"`即可完成问答。\n",
    "> <center><img src=\"https://ml2022.oss-cn-hangzhou.aliyuncs.com/img/image-20250108154120541.png\" alt=\"image-20250108154120541\" style=\"zoom:33%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4fcdae-970f-4f12-8ddd-c082775c3147",
   "metadata": {},
   "source": [
    "#### 1.导入核心关系组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b332ff24-6678-4c9a-abf8-082c04d2f58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import tiktoken\n",
    "\n",
    "from graphrag.query.context_builder.entity_extraction import EntityVectorStoreKey\n",
    "from graphrag.query.indexer_adapters import (\n",
    "    read_indexer_covariates,\n",
    "    read_indexer_entities,\n",
    "    read_indexer_relationships,\n",
    "    read_indexer_reports,\n",
    "    read_indexer_text_units,\n",
    ")\n",
    "from graphrag.query.llm.oai.chat_openai import ChatOpenAI\n",
    "from graphrag.query.llm.oai.embedding import OpenAIEmbedding\n",
    "from graphrag.query.llm.oai.typing import OpenaiApiType\n",
    "from graphrag.query.question_gen.local_gen import LocalQuestionGen\n",
    "from graphrag.query.structured_search.local_search.mixed_context import (\n",
    "    LocalSearchMixedContext,\n",
    ")\n",
    "from graphrag.query.structured_search.local_search.search import LocalSearch\n",
    "from graphrag.vector_stores.lancedb import LanceDBVectorStore"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742bb70a-735c-44fa-b421-5b2af79ba7e2",
   "metadata": {},
   "source": [
    " **与索引器相关的模块**\n",
    "- **`read_indexer_*`**  \n",
    "  从不同的索引文件中读取数据（例如实体、关系、摘要等）。这些模块负责加载索引器生成的数据到 Python 中，供后续搜索或分析使用：\n",
    "  - `read_indexer_covariates`: 读取与数据协变量（附加属性）相关的信息。\n",
    "  - `read_indexer_entities`: 读取从数据中提取的实体。\n",
    "  - `read_indexer_relationships`: 读取实体间的关系。\n",
    "  - `read_indexer_reports`: 读取生成的社区报告摘要。\n",
    "  - `read_indexer_text_units`: 读取切分后的文本单元（TextUnits）。\n",
    "\n",
    "- **`store_entity_semantic_embeddings`**  \n",
    "  将实体的语义嵌入存储到向量数据库中。GraphRAG 用嵌入向量来表示实体间的语义关系。\n",
    "\n",
    " **与 LLM（大型语言模型）相关的模块**\n",
    "- **`ChatOpenAI`**  \n",
    "  一个封装了 OpenAI 聊天模型（如 GPT 系列）的接口，允许你通过编程与 OpenAI 模型交互（如提出问题、获取回答）。\n",
    "\n",
    "- **`OpenAIEmbedding`**  \n",
    "  用于生成文本的嵌入向量的模块，通过调用 OpenAI 的嵌入 API，将文本转换为语义向量表示。\n",
    "\n",
    "- **`OpenaiApiType`**  \n",
    "  定义 OpenAI API 的具体类型，可能包括“聊天模型”、“嵌入模型”等。\n",
    "\n",
    " **与本地搜索相关的模块**\n",
    "- **`LocalSearch`**  \n",
    "  GraphRAG 的本地搜索引擎，专注于通过上下文和邻近信息回答关于特定实体的问题。\n",
    "\n",
    "- **`LocalSearchMixedContext`**  \n",
    "  允许混合使用不同上下文数据（例如实体及其邻居的关系）来丰富本地搜索的结果。\n",
    "\n",
    "- **`LocalQuestionGen`**  \n",
    "  用于在本地搜索中生成问题的模块，帮助生成更相关的问题。\n",
    "\n",
    "  \n",
    " **与向量存储相关的模块**\n",
    "- **`LanceDBVectorStore`**  \n",
    "  GraphRAG 使用的向量存储解决方案之一，支持存储和检索语义嵌入向量。可以快速高效地查找与查询向量最相似的嵌入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bcb1d244-def8-4ffd-9157-368817e509df",
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIR = \"./openl/output\"\n",
    "LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n",
    "\n",
    "COMMUNITY_REPORT_TABLE = \"create_final_community_reports\"\n",
    "ENTITY_TABLE = \"create_final_nodes\"\n",
    "ENTITY_EMBEDDING_TABLE = \"create_final_entities\"\n",
    "RELATIONSHIP_TABLE = \"create_final_relationships\"\n",
    "TEXT_UNIT_TABLE = \"create_final_text_units\"\n",
    "COMMUNITY_LEVEL = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "629e5bb4-30b0-409d-97d2-57f7bcb66e3c",
   "metadata": {},
   "source": [
    " **定义输入目录与文件路径**\n",
    "```python\n",
    "INPUT_DIR = \"./openl/output\"\n",
    "LANCEDB_URI = f\"{INPUT_DIR}/lancedb\"\n",
    "```\n",
    "- **`INPUT_DIR`**：索引器输出文件的存放目录。在这里，索引器输出的路径为 `./openl/output`。\n",
    "- **`LANCEDB_URI`**：存放向量存储（LanceDB 数据库）的目录路径。在 GraphRAG 中，实体嵌入向量通常被存储在 LanceDB 中，以便后续搜索时高效检索。\n",
    "\n",
    " **定义数据表文件名**\n",
    "```python\n",
    "COMMUNITY_REPORT_TABLE = \"create_final_community_reports\"\n",
    "ENTITY_TABLE = \"create_final_nodes\"\n",
    "ENTITY_EMBEDDING_TABLE = \"create_final_entities\"\n",
    "RELATIONSHIP_TABLE = \"create_final_relationships\"\n",
    "COVARIATE_TABLE = \"create_final_covariates\"\n",
    "TEXT_UNIT_TABLE = \"create_final_text_units\"\n",
    "COMMUNITY_LEVEL = 2\n",
    "```\n",
    "- **`COMMUNITY_REPORT_TABLE`**：存储社区报告的文件名。社区报告是基于知识图谱生成的每个社区的摘要。\n",
    "- **`ENTITY_TABLE`**：存储实体的文件名（知识图谱中的节点）。每个节点代表一个实体，例如人、地点或组织。\n",
    "- **`ENTITY_EMBEDDING_TABLE`**：存储实体的嵌入向量。嵌入表示实体的语义信息，用于计算实体之间的相似性。\n",
    "- **`RELATIONSHIP_TABLE`**：存储实体间关系的文件名（知识图谱中的边）。关系表示两个实体之间的交互或连接。\n",
    "- **`TEXT_UNIT_TABLE`**：存储文本单元的文件名（原始文本数据被切分为的小块）。\n",
    "- **`COMMUNITY_LEVEL`**：指定加载的社区层级。社区层级表示知识图谱的聚类粒度（层次化结构中的第2层）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "59f18bb8-251b-43a1-a285-cff12a9183f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity count: 13\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>title</th>\n",
       "      <th>community</th>\n",
       "      <th>level</th>\n",
       "      <th>degree</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35bbb43d-b9f4-4a4d-bd53-7059efc44056</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1535df1-c0e1-4bd4-9b18-49b05918a0aa</td>\n",
       "      <td>1</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9afdfaaf-37d8-4990-9406-a62f37c2d64e</td>\n",
       "      <td>2</td>\n",
       "      <td>CART</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b3651a61-5626-4162-9f1e-432e6d0e9fe5</td>\n",
       "      <td>3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1</td>\n",
       "      <td>4</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id    title  \\\n",
       "0  35bbb43d-b9f4-4a4d-bd53-7059efc44056                  0      ID3   \n",
       "1  f1535df1-c0e1-4bd4-9b18-49b05918a0aa                  1     C4.5   \n",
       "2  9afdfaaf-37d8-4990-9406-a62f37c2d64e                  2     CART   \n",
       "3  b3651a61-5626-4162-9f1e-432e6d0e9fe5                  3  SKLEARN   \n",
       "4  c6f4c391-4f5a-4d99-b8d5-6d4b24f43cc1                  4    NUMPY   \n",
       "\n",
       "   community  level  degree  x  y  \n",
       "0          0      0       9  0  0  \n",
       "1          1      0      11  0  0  \n",
       "2          0      0       8  0  0  \n",
       "3          0      0       3  0  0  \n",
       "4          2      0       4  0  0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read nodes table to get community and degree data\n",
    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
    "entity_embedding_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_EMBEDDING_TABLE}.parquet\")\n",
    "\n",
    "entities = read_indexer_entities(entity_df, entity_embedding_df, COMMUNITY_LEVEL)\n",
    "\n",
    "# load description embeddings to an in-memory lancedb vectorstore\n",
    "# to connect to a remote db, specify url and port values.\n",
    "description_embedding_store = LanceDBVectorStore(\n",
    "    collection_name=\"default-entity-description\",\n",
    ")\n",
    "description_embedding_store.connect(db_uri=LANCEDB_URI)\n",
    "\n",
    "print(f\"Entity count: {len(entity_df)}\")\n",
    "entity_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4aed24dc-cf93-4378-afc0-d10c84c9b977",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relationship count: 29\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>source</th>\n",
       "      <th>target</th>\n",
       "      <th>description</th>\n",
       "      <th>weight</th>\n",
       "      <th>combined_degree</th>\n",
       "      <th>text_unit_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543</td>\n",
       "      <td>0</td>\n",
       "      <td>ID3</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>ID3 and C4.5 are both decision tree algorithms...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>20</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80278b16-6586-491e-af78-f73b39394df9</td>\n",
       "      <td>1</td>\n",
       "      <td>ID3</td>\n",
       "      <td>CART</td>\n",
       "      <td>ID3 and CART are both decision tree algorithms...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>17</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>409e56e1-d48a-4384-ba51-070fd10c58c1</td>\n",
       "      <td>2</td>\n",
       "      <td>ID3</td>\n",
       "      <td>SKLEARN</td>\n",
       "      <td>ID3 is not directly supported by Sklearn, but ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>12</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c54b57c8-c3bb-4aab-b8da-6a1e3ff95726</td>\n",
       "      <td>3</td>\n",
       "      <td>ID3</td>\n",
       "      <td>NUMPY</td>\n",
       "      <td>NumPy is used for entropy calculations in ID3 ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>13</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>651dfca9-7a44-4d11-aed2-c00ce76f936d</td>\n",
       "      <td>4</td>\n",
       "      <td>C4.5</td>\n",
       "      <td>CART</td>\n",
       "      <td>C4.5 and CART are both decision tree algorithm...</td>\n",
       "      <td>17.0</td>\n",
       "      <td>19</td>\n",
       "      <td>[4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  human_readable_id source   target  \\\n",
       "0  a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543                  0    ID3     C4.5   \n",
       "1  80278b16-6586-491e-af78-f73b39394df9                  1    ID3     CART   \n",
       "2  409e56e1-d48a-4384-ba51-070fd10c58c1                  2    ID3  SKLEARN   \n",
       "3  c54b57c8-c3bb-4aab-b8da-6a1e3ff95726                  3    ID3    NUMPY   \n",
       "4  651dfca9-7a44-4d11-aed2-c00ce76f936d                  4   C4.5     CART   \n",
       "\n",
       "                                         description  weight  combined_degree  \\\n",
       "0  ID3 and C4.5 are both decision tree algorithms...    18.0               20   \n",
       "1  ID3 and CART are both decision tree algorithms...    14.0               17   \n",
       "2  ID3 is not directly supported by Sklearn, but ...     2.0               12   \n",
       "3  NumPy is used for entropy calculations in ID3 ...     2.0               13   \n",
       "4  C4.5 and CART are both decision tree algorithm...    17.0               19   \n",
       "\n",
       "                                       text_unit_ids  \n",
       "0  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "1  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "2  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "3  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  \n",
       "4  [4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992a...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relationship_df = pd.read_parquet(f\"{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet\")\n",
    "relationships = read_indexer_relationships(relationship_df)\n",
    "\n",
    "print(f\"Relationship count: {len(relationship_df)}\")\n",
    "relationship_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d9970677-a2c7-4df1-9b32-10deba4aaa98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Report records: 3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>community</th>\n",
       "      <th>parent</th>\n",
       "      <th>level</th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>full_content</th>\n",
       "      <th>rank</th>\n",
       "      <th>rank_explanation</th>\n",
       "      <th>findings</th>\n",
       "      <th>full_content_json</th>\n",
       "      <th>period</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c81a85511b7246ac8f93a15783e26855</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree Algorithms: ID3, C4.5, and CART</td>\n",
       "      <td>This community revolves around key decision tr...</td>\n",
       "      <td># Decision Tree Algorithms: ID3, C4.5, and CAR...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'ID3 is recognized as a class...</td>\n",
       "      <td>{\\n    \"title\": \"Decision Tree Algorithms: ID3...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0c4efb0094a64a3abd1a184b1baab37e</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>C4.5 and Decision Tree Algorithms</td>\n",
       "      <td>This community is centered around the C4.5 dec...</td>\n",
       "      <td># C4.5 and Decision Tree Algorithms\\n\\nThis co...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'C4.5 represents a significan...</td>\n",
       "      <td>{\\n    \"title\": \"C4.5 and Decision Tree Algori...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>983b4cf9540e4ee7820611ff6f6d33ba</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>NumPy and ML_Basic_Function Community</td>\n",
       "      <td>This community is centered around the NumPy li...</td>\n",
       "      <td># NumPy and ML_Basic_Function Community\\n\\nThi...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'NumPy is identified as a Pyt...</td>\n",
       "      <td>{\\n    \"title\": \"NumPy and ML_Basic_Function C...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 id  human_readable_id  community  parent  \\\n",
       "0  c81a85511b7246ac8f93a15783e26855                  0          0      -1   \n",
       "1  0c4efb0094a64a3abd1a184b1baab37e                  1          1      -1   \n",
       "2  983b4cf9540e4ee7820611ff6f6d33ba                  2          2      -1   \n",
       "\n",
       "   level                                          title  \\\n",
       "0      0  Decision Tree Algorithms: ID3, C4.5, and CART   \n",
       "1      0              C4.5 and Decision Tree Algorithms   \n",
       "2      0          NumPy and ML_Basic_Function Community   \n",
       "\n",
       "                                             summary  \\\n",
       "0  This community revolves around key decision tr...   \n",
       "1  This community is centered around the C4.5 dec...   \n",
       "2  This community is centered around the NumPy li...   \n",
       "\n",
       "                                        full_content  rank  \\\n",
       "0  # Decision Tree Algorithms: ID3, C4.5, and CAR...   7.5   \n",
       "1  # C4.5 and Decision Tree Algorithms\\n\\nThis co...   7.5   \n",
       "2  # NumPy and ML_Basic_Function Community\\n\\nThi...   7.5   \n",
       "\n",
       "                                    rank_explanation  \\\n",
       "0  The impact severity rating is high due to the ...   \n",
       "1  The impact severity rating is high due to the ...   \n",
       "2  The impact severity rating is high due to the ...   \n",
       "\n",
       "                                            findings  \\\n",
       "0  [{'explanation': 'ID3 is recognized as a class...   \n",
       "1  [{'explanation': 'C4.5 represents a significan...   \n",
       "2  [{'explanation': 'NumPy is identified as a Pyt...   \n",
       "\n",
       "                                   full_content_json      period  size  \n",
       "0  {\\n    \"title\": \"Decision Tree Algorithms: ID3...  2025-01-08     7  \n",
       "1  {\\n    \"title\": \"C4.5 and Decision Tree Algori...  2025-01-08     4  \n",
       "2  {\\n    \"title\": \"NumPy and ML_Basic_Function C...  2025-01-08     2  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
    "reports = read_indexer_reports(report_df, entity_df, COMMUNITY_LEVEL)\n",
    "\n",
    "print(f\"Report records: {len(report_df)}\")\n",
    "report_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "387142e5-8ad2-4895-9510-05be225a1095",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text unit records: 4\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>text</th>\n",
       "      <th>n_tokens</th>\n",
       "      <th>document_ids</th>\n",
       "      <th>entity_ids</th>\n",
       "      <th>relationship_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992ad...</td>\n",
       "      <td>1</td>\n",
       "      <td>Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7a8436d7970da4b0e5c837b0a3eaaa196e6e2284c7c754...</td>\n",
       "      <td>2</td>\n",
       "      <td>8961919\\n然后即可算出按照如此规则进行数据集划分，最终能够减少的不纯度数值：\\n# ...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1141affdc56bf8f5092b950b57febdedb884673ec6b23d...</td>\n",
       "      <td>3</td>\n",
       "      <td>4.5决策树的基本建模流程\\n作为ID3的改进版算法，C4.5在ID3的基础上进行了三个方面...</td>\n",
       "      <td>1200</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[651dfca9-7a44-4d11-aed2-c00ce76f936d, 3ad66ad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b5df95b6ec7e3aaea3f67db4914e1b2e874f3ac77fc3c8...</td>\n",
       "      <td>4</td>\n",
       "      <td>集划分。\\n\\nC4.5的连续变量处理方法\\nC4.5允许带入连续变量进行建模，并且围绕连续...</td>\n",
       "      <td>558</td>\n",
       "      <td>[2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...</td>\n",
       "      <td>[35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...</td>\n",
       "      <td>[651dfca9-7a44-4d11-aed2-c00ce76f936d, 9ddcf83...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  id  human_readable_id  \\\n",
       "0  4ba5acd798a419708cfe5cc86f1fcb6df907a20b1992ad...                  1   \n",
       "1  7a8436d7970da4b0e5c837b0a3eaaa196e6e2284c7c754...                  2   \n",
       "2  1141affdc56bf8f5092b950b57febdedb884673ec6b23d...                  3   \n",
       "3  b5df95b6ec7e3aaea3f67db4914e1b2e874f3ac77fc3c8...                  4   \n",
       "\n",
       "                                                text  n_tokens  \\\n",
       "0  Lesson 8.3 ID3、C4.5决策树的建模流程\\nID3和C4.5作为的经典决策树算...      1200   \n",
       "1  8961919\\n然后即可算出按照如此规则进行数据集划分，最终能够减少的不纯度数值：\\n# ...      1200   \n",
       "2  4.5决策树的基本建模流程\\n作为ID3的改进版算法，C4.5在ID3的基础上进行了三个方面...      1200   \n",
       "3  集划分。\\n\\nC4.5的连续变量处理方法\\nC4.5允许带入连续变量进行建模，并且围绕连续...       558   \n",
       "\n",
       "                                        document_ids  \\\n",
       "0  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "1  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "2  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "3  [2ef33abc7acb307f736e46c7f20fe87568b4e9bd6311c...   \n",
       "\n",
       "                                          entity_ids  \\\n",
       "0  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "1  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "2  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "3  [35bbb43d-b9f4-4a4d-bd53-7059efc44056, f1535df...   \n",
       "\n",
       "                                    relationship_ids  \n",
       "0  [a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...  \n",
       "1  [a0bbf6be-4a2c-4ad4-9e98-0e26fdf9d543, 80278b1...  \n",
       "2  [651dfca9-7a44-4d11-aed2-c00ce76f936d, 3ad66ad...  \n",
       "3  [651dfca9-7a44-4d11-aed2-c00ce76f936d, 9ddcf83...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_unit_df = pd.read_parquet(f\"{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet\")\n",
    "text_units = read_indexer_text_units(text_unit_df)\n",
    "\n",
    "print(f\"Text unit records: {len(text_unit_df)}\")\n",
    "text_unit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e640a9d-0f1c-4a01-b024-32c52fabb64e",
   "metadata": {},
   "source": [
    "#### 2.设置模型参数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eedad005-aa7c-411d-b776-542a0ec4e04a",
   "metadata": {},
   "source": [
    "- 设置模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "819a2a1c-042f-4813-bfac-edcaaa3fa5b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "sd_api_key = \"YOUR_DeepSeek_API_KEY\"\n",
    "openai_api_key = \"YOUR_DeepSeek_API_KEY\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c41ea76c-e26b-49f4-bd09-5fa69f218318",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_model = \"deepseek-chat\"\n",
    "embedding_model = \"text-embedding-3-large\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d74c5111-cea5-4177-9a4d-d8d4ac2738a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_api_base = \"https://api.deepseek.com\"\n",
    "embedding_model_api_base = \"https://ai.devtool.tech/proxy/v1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9b4ba24a-1a28-4893-9734-18a3f71c3806",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(\n",
    "    api_key=sd_api_key,\n",
    "    model=llm_model,\n",
    "    api_base=llm_api_base,\n",
    "    api_type=OpenaiApiType.OpenAI,  \n",
    "    max_retries=20,\n",
    ")\n",
    "\n",
    "token_encoder = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "text_embedder = OpenAIEmbedding(\n",
    "    api_key=openai_api_key,\n",
    "    api_base=embedding_model_api_base,\n",
    "    api_type=OpenaiApiType.OpenAI,\n",
    "    model=embedding_model,\n",
    "    deployment_name=embedding_model,\n",
    "    max_retries=20,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0436ae5e-2f21-44ae-b1a1-ef43ac90cb83",
   "metadata": {},
   "source": [
    "**初始化 Chat 模型**\n",
    "- **`ChatOpenAI`**：\n",
    "  - 用于与 OpenAI 或 Azure OpenAI 的聊天模型（如 GPT 系列）交互。\n",
    "  - **参数说明**：\n",
    "    - **`api_key`**：访问 OpenAI 或 Azure OpenAI 的密钥。\n",
    "    - **`model`**：指定 LLM 模型名称（如 `gpt-4`）。\n",
    "    - - **`api_base`**：API 的基础路径，可以填写反向代理地址\n",
    "    - **`api_type`**：API 类型，可以是：\n",
    "      - **`OpenaiApiType.OpenAI`**：OpenAI 直连服务。\n",
    "      - **`OpenaiApiType.AzureOpenAI`**：Azure OpenAI 服务。\n",
    "    - **`max_retries`**：最大重试次数，以处理网络或服务中断。\n",
    "\n",
    "**初始化文本分词器**\n",
    "- **`tiktoken`**：\n",
    "  - 用于对输入文本进行分词或令牌化操作。\n",
    "  - **`cl100k_base`**：一种令牌编码方式，适用于 GPT 系列模型（如 `gpt-3.5-turbo` 和 `gpt-4`）。\n",
    "  - **用途**：\n",
    "    - 计算输入文本的令牌数量，确保不超过模型的上下文窗口限制。\n",
    "\n",
    "**初始化嵌入生成器**\n",
    "- **`OpenAIEmbedding`**：\n",
    "  - 用于生成文本的嵌入向量。\n",
    "  - **参数说明**：\n",
    "    - **`api_key`**：用于访问 OpenAI 嵌入 API 的密钥。\n",
    "    - **`api_base`**：API 的基础路径，可以填写反向代理地址\n",
    "    - **`api_type`**：API 类型，与 LLM 相同。\n",
    "    - **`model`**：指定嵌入模型名称，例如 `text-embedding-ada-002`。\n",
    "    - **`deployment_name`**：Azure 模型部署名称（如果使用 Azure）。\n",
    "    - **`max_retries`**：最大重试次数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f98f060-a420-497e-b8b9-4746b9bdb51f",
   "metadata": {},
   "source": [
    "#### 3.构建LocalSearch（本地搜索）搜索引擎并进行问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0589d65b-03d1-47db-82b9-e480a11e5020",
   "metadata": {},
   "source": [
    "- **创建LocalSearch上下文构建器**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1c03b562-c7a2-4d5d-8e49-4bd00f7232a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_builder = LocalSearchMixedContext(\n",
    "    community_reports=reports,\n",
    "    text_units=text_units,\n",
    "    entities=entities,\n",
    "    relationships=relationships,\n",
    "    covariates=None,\n",
    "    entity_text_embeddings=description_embedding_store,\n",
    "    embedding_vectorstore_key=EntityVectorStoreKey.ID,  \n",
    "    text_embedder=text_embedder,\n",
    "    token_encoder=token_encoder,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee4920c-9f2f-48ff-bcfa-f5ff29886f41",
   "metadata": {},
   "source": [
    "&emsp;&emsp;本段代码的主要功能是**创建本地搜索的上下文构建器（Context Builder）**。上下文构建器的作用是整合所有加载的数据（如社区报告、文本单元、实体、关系等），为后续的本地搜索提供语义丰富的上下文。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83d1c81-7a2d-4fe1-82c1-d32dd411675a",
   "metadata": {},
   "source": [
    " **参数解析**\n",
    "1. **`community_reports`**\n",
    "   - 传入之前加载的社区报告数据（`reports`）。\n",
    "   - 用于补充社区级别的背景信息，在本地搜索中提供更多上下文。\n",
    "\n",
    "2. **`text_units`**\n",
    "   - 传入已加载的文本单元数据（`text_units`）。\n",
    "   - 提供来自原始文档的小块文本，用于回答具体问题或补充答案的上下文。\n",
    "\n",
    "3. **`entities`**\n",
    "   - 传入知识图谱中的实体（`entities`）。\n",
    "   - 提供实体的详细信息和属性，帮助本地搜索定位特定实体。\n",
    "\n",
    "4. **`relationships`**\n",
    "   - 传入知识图谱中的关系（`relationships`）。\n",
    "   - 用于描述实体之间的交互和连接，有助于构建更复杂的答案。\n",
    "\n",
    "5. **`covariates`**\n",
    "   - 传入协变量数据（`covariates`），如果索引阶段未启用协变量，设置为 `None`。\n",
    "   - **用途**：\n",
    "     - 提供附加声明性元数据或上下文信息。\n",
    "     - 如果协变量未启用，仍可正常运行本地搜索，但答案可能少一些附加的语义信息。\n",
    "\n",
    "6. **`entity_text_embeddings`**\n",
    "   - 传入实体嵌入的存储对象（`description_embedding_store`）。\n",
    "   - **用途**：\n",
    "     - 将实体的文本嵌入存储在向量数据库中。\n",
    "     - 用于通过语义搜索快速找到相关实体。\n",
    "\n",
    "7. **`embedding_vectorstore_key`**\n",
    "   - 设置嵌入向量存储的键类型。\n",
    "   - **`EntityVectorStoreKey.ID`**：\n",
    "     - 如果向量存储中的实体是按 ID 存储的，使用此键。\n",
    "   - **`EntityVectorStoreKey.TITLE`**：\n",
    "     - 如果向量存储按实体标题存储，则改为设置为 `TITLE`。\n",
    "\n",
    "8. **`text_embedder`**\n",
    "   - 传入嵌入生成器对象（`text_embedder`）。\n",
    "   - 用于生成文本或问题的语义嵌入，支持与存储的实体嵌入进行相似度比较。\n",
    "\n",
    "9. **`token_encoder`**\n",
    "   - 传入分词器（`token_encoder`）。\n",
    "   - **用途**：\n",
    "     - 确保生成的上下文不会超过 LLM 的上下文窗口限制。\n",
    "     - 有助于在构建查询时分配合理的上下文。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "475bb76a-f3f7-46ee-84b4-0408e43fcd85",
   "metadata": {},
   "source": [
    "**上下文构建器的作用**\n",
    "上下文构建器是本地搜索的核心组件，它负责将加载的数据整合成结构化的上下文，包括：\n",
    "1. **结合社区级摘要（Community Reports）**：\n",
    "   - 提供更高层次的背景信息，用于回答主题性问题。\n",
    "\n",
    "2. **整合文本单元（Text Units）**：\n",
    "   - 使用来自原始文档的文本片段，为问题构建直接的上下文。\n",
    "\n",
    "3. **集成知识图谱的实体和关系**：\n",
    "   - 为本地搜索中的实体相关问题提供详细的语义信息。\n",
    "\n",
    "4. **利用嵌入和分词**：\n",
    "   - 嵌入存储和分词器确保上下文生成语义准确，符合 LLM 的上下文限制。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e909696c-2517-498e-9530-2cc23f28e314",
   "metadata": {},
   "source": [
    "- **创建本地搜索引擎参数组**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ce855ef1-4bf3-425d-b4f0-deab58890f8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_context_params = {\n",
    "    \"text_unit_prop\": 0.5,\n",
    "    \"community_prop\": 0.1,\n",
    "    \"conversation_history_max_turns\": 5,\n",
    "    \"conversation_history_user_turns_only\": True,\n",
    "    \"top_k_mapped_entities\": 10,\n",
    "    \"top_k_relationships\": 10,\n",
    "    \"include_entity_rank\": True,\n",
    "    \"include_relationship_weight\": True,\n",
    "    \"include_community_rank\": True,\n",
    "    \"return_candidate_context\": True,\n",
    "    \"embedding_vectorstore_key\": EntityVectorStoreKey.ID,  \n",
    "    \"max_tokens\": 12_000, \n",
    "}\n",
    "\n",
    "llm_params = {\n",
    "    \"max_tokens\": 2_000, \n",
    "    \"temperature\": 0.0,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fc1e1f0-0ba4-43ff-8c0b-402c38c74c25",
   "metadata": {},
   "source": [
    "**参数解析**：\n",
    "- **`text_unit_prop`**：\n",
    "  - 上下文窗口中分配给文本单元（Text Units）的比例（50%）。\n",
    "  - 控制原始文档中的文本内容在上下文中的权重。\n",
    "\n",
    "- **`community_prop`**：\n",
    "  - 上下文窗口中分配给社区报告（Community Reports）的比例（10%）。\n",
    "  - 用于为回答提供更高层次的社区背景。\n",
    "\n",
    "- **`conversation_history_max_turns`**：\n",
    "  - 上下文中包含的最大对话轮数（5轮）。\n",
    "  - 控制是否包括之前的对话历史，以及保留的对话轮数。\n",
    "\n",
    "- **`conversation_history_user_turns_only`**：\n",
    "  - 如果为 `True`，则只包括用户的提问，而忽略模型的回答。\n",
    "\n",
    "- **`top_k_mapped_entities`**：\n",
    "  - 从嵌入向量存储中检索的相关实体的数量（10个）。\n",
    "\n",
    "- **`top_k_relationships`**：\n",
    "  - 检索到上下文窗口中的最大关系数（10个）。\n",
    "\n",
    "- **`include_entity_rank`**：\n",
    "  - 是否在上下文中包含实体的排名（默认是根据实体的节点度数）。\n",
    "\n",
    "- **`include_relationship_weight`**：\n",
    "  - 是否在上下文中包含关系权重（例如，表示关系的重要性）。\n",
    "\n",
    "- **`include_community_rank`**：\n",
    "  - 是否在上下文中包含社区排名。这里设置为 `False`。\n",
    "\n",
    "- **`return_candidate_context`**：\n",
    "  - 如果为 `True`，返回一个数据帧集合，包含所有可能相关的实体、关系和协变量记录。这些记录的上下文中会有一列 `in_context` 表示是否被纳入上下文窗口。\n",
    "\n",
    "- **`embedding_vectorstore_key`**：\n",
    "  - 嵌入向量存储的键类型。\n",
    "  - 如果向量存储的标识符是实体标题而非 ID，则设置为 `EntityVectorStoreKey.TITLE`。\n",
    "\n",
    "- **`max_tokens`**：\n",
    "  - 上下文窗口允许的最大令牌数（12,000个）。\n",
    "  - 需要根据所用 LLM 的令牌限制调整（例如，GPT-4 支持 8,192 或 32,768）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6be799ab-a5ca-43bb-9bc3-5397b5e264a7",
   "metadata": {},
   "source": [
    "- **创建本地搜索引擎**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3c98d816-e056-4b6a-8a94-d57a640c0862",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_engine = LocalSearch(\n",
    "    llm=llm,\n",
    "    context_builder=context_builder,\n",
    "    token_encoder=token_encoder,\n",
    "    llm_params=llm_params,\n",
    "    context_builder_params=local_context_params,\n",
    "    response_type=\"multiple paragraphs\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3fb4d3-19df-4c57-8b8a-d7d77a7ceabc",
   "metadata": {},
   "source": [
    "**参数解析**：\n",
    "- **`llm`**：\n",
    "  - 已初始化的 LLM 对象（`ChatOpenAI`），用于生成答案。\n",
    "\n",
    "- **`context_builder`**：\n",
    "  - 上一段代码中创建的上下文构建器（`LocalSearchMixedContext`），负责生成语义丰富的上下文。\n",
    "\n",
    "- **`token_encoder`**：\n",
    "  - 分词器，用于计算上下文和生成回答的令牌数，确保不会超过 LLM 的限制。\n",
    "\n",
    "- **`llm_params`**：\n",
    "  - 用于 LLM 的参数，包括生成答案时的令牌数和温度。\n",
    "\n",
    "- **`context_builder_params`**：\n",
    "  - 上下文构建器的配置，控制上下文中每种数据的比例和数量限制。\n",
    "\n",
    "- **`response_type`**：\n",
    "  - 指定回答的格式。\n",
    "  - 此处为 `\"multiple paragraphs\"`，表示生成的回答以多段文字形式呈现。\n",
    "  - 可以根据需求设置为 `\"single paragraph\"` 或 `\"prioritized list\"` 等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f358b2f5-0769-45ec-888b-8fe850d97387",
   "metadata": {},
   "source": [
    "- 基于本地搜索的问答流程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1e19c2e1-41c9-45d3-98c8-0dffd3432e8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = await search_engine.asearch(\"请帮我介绍下ID3决策树算法\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d7c67e53-050a-496b-809c-2e44ec6c8512",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### ID3决策树算法概述\n",
       "\n",
       "ID3（Iterative Dichotomiser 3）是一种经典的决策树算法，主要用于分类任务。它通过信息熵（information entropy）来评估数据集划分的纯度，并选择信息增益（information gain）最大的属性进行节点分裂。ID3的核心思想是通过递归地选择最佳属性来构建决策树，从而实现对数据的分类 [Data: Entities (0); Reports (0); Relationships (0)].\n",
       "\n",
       "#### 信息熵与信息增益\n",
       "ID3使用信息熵来衡量数据集的混乱程度。信息熵越低，数据集的纯度越高。信息增益则是通过计算分裂前后信息熵的差值来评估某个属性对数据集划分的效果。ID3在每一步选择信息增益最大的属性进行分裂，从而逐步构建决策树 [Data: Entities (0); Reports (0); Relationships (27)].\n",
       "\n",
       "#### 数据集划分与离散变量处理\n",
       "ID3只能处理离散变量，无法直接处理连续变量。如果数据集中包含连续变量，需要先对其进行离散化处理（如分箱）。ID3通过一次展开一列的方式对数据集进行划分，这与CART算法中寻找切分点的方式不同。例如，在某个数据集中，ID3会根据`AGE`列的不同取值将数据集划分为多个子集，然后计算每个子集的信息熵和信息增益 [Data: Sources (0, 1); Entities (6, 7, 8, 9); Relationships (10, 11, 12, 13)].\n",
       "\n",
       "#### ID3的局限性\n",
       "尽管ID3在决策树算法的发展中具有重要地位，但它也存在一些局限性：\n",
       "1. **无法处理连续变量**：ID3只能处理离散变量，连续变量需要先进行离散化处理 [Data: Entities (0); Reports (0)].\n",
       "2. **倾向于选择多分类属性**：ID3在选择分裂属性时，倾向于选择分类水平较多的属性，这可能导致模型过拟合 [Data: Sources (1); Entities (0)].\n",
       "3. **缺乏剪枝机制**：ID3没有内置的剪枝机制来防止过拟合，这使得模型在训练数据上表现良好，但在测试数据上可能表现不佳 [Data: Sources (1); Entities (0)].\n",
       "\n",
       "#### ID3与C4.5、CART的关系\n",
       "ID3是决策树算法的基础，后续的C4.5和CART算法在ID3的基础上进行了改进。C4.5引入了连续变量处理、剪枝机制和信息值（information value）等改进，而CART则支持分类和回归任务，并采用二分法进行数据集划分 [Data: Entities (1, 2); Relationships (0, 1, 4, 14)].\n",
       "\n",
       "### 总结\n",
       "ID3作为决策树算法的奠基者，通过信息熵和信息增益构建分类模型，尽管存在一些局限性，但其核心思想为后续算法的发展提供了重要基础。C4.5和CART等算法在ID3的基础上进行了优化，进一步提升了决策树模型的性能和适用性 [Data: Entities (0, 1, 2); Reports (0, 1); Relationships (0, 1, 4, 14)]."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(result.response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "05fca1b9-e67b-4d9c-a00b-98cd3b7e851e",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = await search_engine.asearch(\"请帮我介绍下ID3决策树算法\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2272b5f9-396b-42b1-8f02-2f7c764d0e62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### ID3决策树算法概述\n",
       "\n",
       "ID3（Iterative Dichotomiser 3）是一种经典的决策树算法，主要用于分类任务。它通过信息熵（Information Entropy）来评估数据集划分的纯度，并选择信息增益（Information Gain）最大的属性进行节点分裂。ID3算法是决策树算法家族中的基础算法，后续的C4.5和CART算法都在其基础上进行了改进和扩展 [Data: Entities (0); Reports (0); Relationships (0, 23)]。\n",
       "\n",
       "---\n",
       "\n",
       "### ID3的核心原理\n",
       "\n",
       "ID3的核心思想是通过递归地选择信息增益最大的属性来构建决策树。具体步骤如下：\n",
       "\n",
       "1. **信息熵计算**：ID3使用信息熵来衡量数据集的纯度。信息熵越低，数据集的纯度越高。对于一个数据集，信息熵的计算公式为：\n",
       "   \\[\n",
       "   \\text{Entropy}(S) = -\\sum_{i=1}^{n} p_i \\log_2(p_i)\n",
       "   \\]\n",
       "   其中，\\( p_i \\) 是数据集中第 \\( i \\) 类样本的比例 [Data: Sources (0, 1)].\n",
       "\n",
       "2. **信息增益计算**：信息增益用于衡量某个属性对数据集划分的贡献。信息增益越大，说明该属性对分类的贡献越大。信息增益的计算公式为：\n",
       "   \\[\n",
       "   \\text{Gain}(S, A) = \\text{Entropy}(S) - \\sum_{v \\in \\text{Values}(A)} \\frac{|S_v|}{|S|} \\text{Entropy}(S_v)\n",
       "   \\]\n",
       "   其中，\\( S_v \\) 是属性 \\( A \\) 取值为 \\( v \\) 时的子集 [Data: Sources (1); Entities (11)].\n",
       "\n",
       "3. **节点分裂**：ID3选择信息增益最大的属性作为当前节点的分裂属性，并根据该属性的不同取值将数据集划分为多个子集。这一过程递归进行，直到所有子集的纯度达到要求或没有更多属性可用 [Data: Sources (0, 1)].\n",
       "\n",
       "---\n",
       "\n",
       "### ID3的局限性\n",
       "\n",
       "尽管ID3算法在决策树领域具有重要地位，但它也存在一些明显的局限性：\n",
       "\n",
       "1. **仅支持离散变量**：ID3只能处理离散型变量，无法直接处理连续型变量。如果数据集中包含连续型变量，需要先进行离散化处理 [Data: Entities (0); Sources (0)].\n",
       "\n",
       "2. **倾向于选择多分类属性**：ID3在选择分裂属性时，倾向于选择分类水平较多的属性，这可能导致模型过拟合 [Data: Entities (0); Sources (1)].\n",
       "\n",
       "3. **缺乏剪枝机制**：ID3没有内置的剪枝机制，无法有效防止过拟合 [Data: Entities (0); Sources (1)].\n",
       "\n",
       "4. **不支持回归任务**：ID3仅适用于分类任务，无法处理回归问题 [Data: Entities (0)].\n",
       "\n",
       "---\n",
       "\n",
       "### ID3与C4.5和CART的关系\n",
       "\n",
       "ID3是决策树算法家族中的基础算法，后续的C4.5和CART算法在其基础上进行了改进：\n",
       "\n",
       "- **C4.5**：C4.5在ID3的基础上引入了连续变量处理、剪枝机制和信息增益比（Gain Ratio）等改进，解决了ID3的许多局限性 [Data: Entities (1); Relationships (0, 23)].\n",
       "\n",
       "- **CART**：CART（Classification and Regression Trees）进一步扩展了决策树的能力，支持分类和回归任务，并采用二分法进行节点分裂 [Data: Entities (2); Relationships (1, 4)].\n",
       "\n",
       "---\n",
       "\n",
       "### 总结\n",
       "\n",
       "ID3算法作为决策树领域的开创性算法，奠定了决策树模型的基础。尽管它存在一些局限性，但其核心思想（如信息熵和信息增益）在后续的C4.5和CART算法中得到了继承和发展。ID3算法在分类任务中仍然具有一定的应用价值，尤其是在处理离散型数据时 [Data: Entities (0); Reports (0); Relationships (0, 23)]."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(result.response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9050cc10-22ad-4c57-a3c6-3e4ac8b2d34e",
   "metadata": {},
   "source": [
    "#### 4.构建GlobalSearch（全局搜索）搜索引擎并进行问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84a1e06b-8711-4b74-b50b-9bfc6ed1dec1",
   "metadata": {},
   "source": [
    "- 导入知识图谱相关内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c8f0d9fd-bff7-4916-9e95-e8e75ae28264",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tiktoken\n",
    "\n",
    "from graphrag.query.indexer_adapters import (\n",
    "    read_indexer_communities,\n",
    "    read_indexer_entities,\n",
    "    read_indexer_reports,\n",
    ")\n",
    "from graphrag.query.llm.oai.chat_openai import ChatOpenAI\n",
    "from graphrag.query.llm.oai.typing import OpenaiApiType\n",
    "from graphrag.query.structured_search.global_search.community_context import (\n",
    "    GlobalCommunityContext,\n",
    ")\n",
    "from graphrag.query.structured_search.global_search.search import GlobalSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7583ed4c-67ce-438f-8393-887177fec2f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMMUNITY_TABLE = \"create_final_communities\"\n",
    "COMMUNITY_REPORT_TABLE = \"create_final_community_reports\"\n",
    "ENTITY_TABLE = \"create_final_nodes\"\n",
    "ENTITY_EMBEDDING_TABLE = \"create_final_entities\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5045668b-c3e3-46b0-a45f-2a88c936d73b",
   "metadata": {},
   "outputs": [],
   "source": [
    "community_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet\")\n",
    "entity_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_TABLE}.parquet\")\n",
    "report_df = pd.read_parquet(f\"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet\")\n",
    "entity_embedding_df = pd.read_parquet(f\"{INPUT_DIR}/{ENTITY_EMBEDDING_TABLE}.parquet\")\n",
    "\n",
    "communities = read_indexer_communities(community_df, entity_df, report_df)\n",
    "reports = read_indexer_reports(report_df, entity_df, COMMUNITY_LEVEL)\n",
    "entities = read_indexer_entities(entity_df, entity_embedding_df, COMMUNITY_LEVEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dd71b0f-14cb-4d2c-bab5-8e6764e9a28b",
   "metadata": {},
   "source": [
    "- 创建Global Search模式的上下文构建器 (context_builder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "452aaf7a-e0f2-4999-a1fe-37ec83ef38a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_builder = GlobalCommunityContext(\n",
    "    community_reports=reports,\n",
    "    communities=communities,\n",
    "    entities=entities,  \n",
    "    token_encoder=token_encoder,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7245dc9c-51a9-48f4-b40b-af87712a8ffb",
   "metadata": {},
   "source": [
    "代码解释如下：\n",
    "1. **`GlobalCommunityContext`**:\n",
    "   - 这是一个为 **Global Search** 模式创建的上下文构建器。它负责从不同的数据源（如社区报告、实体、社区信息等）中提取相关数据，并构建一个用于查询的大范围上下文。\n",
    "   - 这个构建器帮助为整个文档集合（而不是某个特定实体）构建一个全局的知识背景，以便模型能够对广泛的问题做出响应。\n",
    "\n",
    "2. **`community_reports=reports`**:\n",
    "   - 这里传入的是之前从文件中读取的 `reports` 数据（也就是 `COMMUNITY_REPORT_TABLE` 表格的数据）。\n",
    "   - 这些报告包含了关于不同社区的详细信息，如主题、摘要、影响力、发现等。\n",
    "   - 在全局搜索中，这些社区报告有助于构建整个数据集的高层次概览。\n",
    "\n",
    "3. **`communities=communities`**:\n",
    "   - `communities` 数据包含社区的结构和信息。在图谱中，社区可能代表了不同的主题、领域或相关性较高的实体群体。\n",
    "   - 这部分数据通常用于在全局搜索时进行分组和排名，比如通过社群的重要性或与查询的相关性来排序。\n",
    "\n",
    "4. **`entities=entities`**:\n",
    "   - `entities` 是从之前的索引步骤中提取的实体数据（来自 `ENTITY_TABLE` 表）。\n",
    "   - 这些实体（如人物、地点、事件等）可以用来扩展全局搜索的范围。它们提供了具体的名词、对象和概念，有助于为全局查询提供上下文。\n",
    "\n",
    "   - **注意：** 如果不希望在全局搜索中使用社区权重来影响排名（例如，不考虑某个社区对某个实体的影响），可以将 `entities` 参数设为 `None`。\n",
    "\n",
    "5. **`token_encoder=token_encoder`**:\n",
    "   - `token_encoder` 是用于对文本进行编码的工具。它将文本转化为可以输入到模型中的 token 序列。\n",
    "   - 在之前的代码中，我们已经看到 `token_encoder` 是通过 `tiktoken.get_encoding(\"cl100k_base\")` 获取的，它的作用是把文本切分为一定的 token 数量，以便在模型中使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e27b8f3c-6bec-48f1-9a51-7f7c36ca2a91",
   "metadata": {},
   "source": [
    "- 配置全局搜索参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "05b9f3a1-a0f2-4cf6-8f63-7f3dbd0c5d7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_builder_params = {\n",
    "    \"use_community_summary\": False,  \n",
    "    \"shuffle_data\": True,\n",
    "    \"include_community_rank\": True,\n",
    "    \"min_community_rank\": 0,\n",
    "    \"community_rank_name\": \"rank\",\n",
    "    \"include_community_weight\": True,\n",
    "    \"community_weight_name\": \"occurrence weight\",\n",
    "    \"normalize_community_weight\": True,\n",
    "    \"max_tokens\": 12_000,  \n",
    "    \"context_name\": \"Reports\",\n",
    "}\n",
    "\n",
    "map_llm_params = {\n",
    "    \"max_tokens\": 1000,\n",
    "    \"temperature\": 0.0,\n",
    "    \"response_format\": {\"type\": \"json_object\"},\n",
    "}\n",
    "\n",
    "reduce_llm_params = {\n",
    "    \"max_tokens\": 2000, \n",
    "    \"temperature\": 0.0,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5d387dd-29ca-49b5-9f77-6e38d8cbc06d",
   "metadata": {},
   "source": [
    "- 构建全局搜索引擎"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9f137781-7dc0-4594-b243-3db58c707b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_engine = GlobalSearch(\n",
    "    llm=llm,\n",
    "    context_builder=context_builder,\n",
    "    token_encoder=token_encoder,\n",
    "    max_data_tokens=12_000,  \n",
    "    map_llm_params=map_llm_params,\n",
    "    reduce_llm_params=reduce_llm_params,\n",
    "    allow_general_knowledge=False, \n",
    "    json_mode=True,  \n",
    "    context_builder_params=context_builder_params,\n",
    "    concurrent_coroutines=32,\n",
    "    response_type=\"multiple paragraphs\",  \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13280544-becb-4279-bffc-5d654cc1eb4c",
   "metadata": {},
   "source": [
    "这里创建了一个 **全局搜索（Global Search）** 引擎，利用前面配置的上下文构建器 (`context_builder`)、语言模型 (`llm`)、以及其它相关的参数进行搜索。对应参数详解如下：\n",
    "\n",
    "- **`llm`**:\n",
    "   - 这个参数传入的是已经配置好的语言模型（LLM），即你之前定义的 `ChatOpenAI` 模型。全局搜索引擎将使用这个模型来生成回答。\n",
    "\n",
    "- **`context_builder`**:\n",
    "   - 上下文构建器，之前我们已经讨论了 `GlobalCommunityContext`，它负责根据社区报告、社区等信息来构建查询的上下文。\n",
    "\n",
    "- **`token_encoder`**:\n",
    "   - `token_encoder` 是用来处理文本的编码工具，通常是一个模型专用的 tokenizer。在这里，我们使用了 `tiktoken.get_encoding(\"cl100k_base\")`，它是 OpenAI 模型的一个编码器，用来将文本转化为 token 格式。\n",
    "\n",
    "- **`max_data_tokens`**:\n",
    "   - 最大数据 token 数量。它控制了模型可以处理的最大上下文大小。你可以根据实际使用的模型的最大 token 限制来设置（例如，如果模型最大 token 限制是 8K，则可以设置为 5000，留一些余量）。这个设置控制了全局搜索时在上下文窗口内所使用的最大 token 数。\n",
    "\n",
    "- **`map_llm_params`** 和 **`reduce_llm_params`**:\n",
    "   - 这两个参数分别传入了映射和归约阶段的 LLM 配置（之前我们已经详细分析过这些参数）。这些参数会影响 LLM 在不同阶段生成内容的方式（例如，`max_tokens` 和 `temperature` 等）。\n",
    "\n",
    "- **`allow_general_knowledge`**:\n",
    "   - 这个参数控制是否允许模型在回答中加入通用知识。如果设置为 `True`，LLM 会尝试将 **外部知识** 加入到查询的结果中。这对于需要广泛知识背景的任务可能有帮助，但也有可能导致模型生成 **虚假信息**（hallucinations）。为了避免这个问题，默认值设置为 `False`。\n",
    "\n",
    "- **`json_mode`**:\n",
    "   - 这个参数控制结果的格式。如果设置为 `True`，LLM 会生成结构化的 **JSON 格式** 输出。如果设置为 `False`，则返回的是更自由形式的文本回答。对于结构化数据的处理，通常使用 JSON 格式。\n",
    "\n",
    "- **`context_builder_params`**:\n",
    "   - 这是传入给上下文构建器的参数，用来进一步配置如何构建查询上下文（例如是否使用社区简要摘要、是否包含社区排名等）。我们在之前的代码分析中已经详细介绍了这些参数。\n",
    "\n",
    "- **`concurrent_coroutines`**:\n",
    "   - 这个参数控制并发协程的数量。全局搜索引擎支持并发处理多个查询，如果你需要同时处理多个请求，可以增加这个值。比如设置为 `32`，意味着最多可以同时处理 32 个查询。\n",
    "\n",
    "- **`response_type`**:\n",
    "   - 这个参数定义了模型生成的响应的类型和格式。它是一个自由文本的描述，指明返回的内容应该是什么样的格式。在这里，`\"multiple paragraphs\"` 表示模型会生成多段文字的回答，适合长篇的说明或报告。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ec43e7c-92a2-40e2-8db0-edd6a090ab7e",
   "metadata": {},
   "source": [
    "- 执行全局搜索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "3970ef08-eb09-4bb9-bd9e-47357cfd6bda",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = await search_engine.asearch(\"请帮我介绍下ID3决策树算法\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5f0a182a-f2f0-4698-9497-212d7719e88b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "ID3（Iterative Dichotomiser 3）是一种经典的决策树算法，主要用于分类任务。它通过信息熵（information entropy）来评估分裂的纯度，并专注于处理离散变量。信息熵的使用使得ID3能够有效地根据最具信息量的属性对数据集进行划分，从而构建决策树 [Data: Reports (0)]。\n",
       "\n",
       "### 核心原理\n",
       "ID3的核心在于其使用信息熵来衡量数据集的混乱程度，并选择能够最大程度减少熵的属性进行分裂。这种方法确保了决策树在每一步都能选择最优的分裂点，从而提高分类的准确性。然而，ID3仅适用于离散变量，无法直接处理连续变量 [Data: Reports (0)]。\n",
       "\n",
       "### 局限性\n",
       "尽管ID3在决策树算法的发展中起到了奠基作用，但它也存在一些明显的局限性。首先，ID3容易过拟合，尤其是在处理复杂数据集时。其次，它无法直接处理连续变量，这限制了其在实际应用中的灵活性 [Data: Reports (0)]。\n",
       "\n",
       "### 后续发展\n",
       "ID3的局限性促使了后续算法的改进。例如，C4.5算法在ID3的基础上引入了对连续变量的处理、剪枝技术以及改进的属性选择标准，使其成为一个更强大且通用的算法。CART（Classification and Regression Trees）也是ID3的重要后继者，进一步扩展了决策树的应用范围 [Data: Reports (0)]。\n",
       "\n",
       "### 总结\n",
       "ID3作为决策树算法的先驱，为后续算法的发展奠定了重要基础。尽管它存在一些局限性，但其核心思想——通过信息熵进行分裂——仍然是现代决策树算法的核心组成部分。通过C4.5和CART等算法的改进，ID3的局限性得到了有效解决，进一步推动了决策树技术在机器学习领域的广泛应用 [Data: Reports (0)]。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(result.response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "f5030586-7682-4155-b027-09a247261994",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = await search_engine.asearch(\"请帮我介绍下ID3决策树算法\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "7c0c8ea7-88ae-4fad-a271-5d090d33c71d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "ID3（Iterative Dichotomiser 3）是一种经典的决策树算法，主要用于分类任务。它通过信息熵（information entropy）来评估数据分割的纯度，并专注于处理离散变量。信息熵的使用是ID3算法构建决策树的核心方法，通过计算信息增益来选择最佳的分割属性，从而逐步构建决策树 [Data: Reports (0)]。\n",
       "\n",
       "### 核心原理\n",
       "ID3算法的核心在于利用信息熵来衡量数据集的混乱程度。信息熵越低，数据集的纯度越高。算法通过计算每个属性的信息增益，选择能够最大程度减少不确定性的属性作为分割节点。这一过程递归进行，直到所有数据被正确分类或达到停止条件 [Data: Reports (0)]。\n",
       "\n",
       "### 优点与局限性\n",
       "ID3算法在决策树的发展中具有重要的历史意义，但其也存在一些局限性。首先，ID3只能处理离散变量，无法直接处理连续变量。其次，ID3容易出现过拟合问题，尤其是在数据集较小或噪声较多的情况下。这些局限性促使了后续算法的改进 [Data: Reports (0, 1)]。\n",
       "\n",
       "### 后续发展\n",
       "ID3的局限性催生了更先进的决策树算法，如C4.5和CART。这些算法在ID3的基础上进行了改进，例如引入了连续变量的处理能力、剪枝技术（pruning）以及更复杂的属性选择标准。C4.5和CART的改进不仅克服了ID3的不足，还进一步提升了决策树的性能和适用性 [Data: Reports (0, 1)]。\n",
       "\n",
       "### 总结\n",
       "ID3算法作为决策树领域的开创性工作，奠定了后续算法发展的基础。尽管它存在一些局限性，但其核心思想——利用信息熵进行数据分割——仍然是决策树算法的重要组成部分。通过了解ID3，我们可以更好地理解现代决策树算法的演进过程及其背后的原理 [Data: Reports (0, 1)]。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(result.response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b5ba9d-65f2-49b2-b03d-4b0249999ee3",
   "metadata": {},
   "source": [
    "- 查看全局搜索调用的社区报告"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "27385ac6-3fe4-444a-8716-09092a992292",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>occurrence weight</th>\n",
       "      <th>content</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>C4.5 and Decision Tree Algorithms</td>\n",
       "      <td>1.00</td>\n",
       "      <td># C4.5 and Decision Tree Algorithms\\n\\nThis co...</td>\n",
       "      <td>7.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree Algorithms: ID3, C4.5, and CART</td>\n",
       "      <td>1.00</td>\n",
       "      <td># Decision Tree Algorithms: ID3, C4.5, and CAR...</td>\n",
       "      <td>7.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>NumPy and ML_Basic_Function Community</td>\n",
       "      <td>0.25</td>\n",
       "      <td># NumPy and ML_Basic_Function Community\\n\\nThi...</td>\n",
       "      <td>7.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  id                                          title  occurrence weight  \\\n",
       "0  1              C4.5 and Decision Tree Algorithms               1.00   \n",
       "1  0  Decision Tree Algorithms: ID3, C4.5, and CART               1.00   \n",
       "2  2          NumPy and ML_Basic_Function Community               0.25   \n",
       "\n",
       "                                             content  rank  \n",
       "0  # C4.5 and Decision Tree Algorithms\\n\\nThis co...   7.5  \n",
       "1  # Decision Tree Algorithms: ID3, C4.5, and CAR...   7.5  \n",
       "2  # NumPy and ML_Basic_Function Community\\n\\nThi...   7.5  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.context_data[\"reports\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b4ba8d8e-4173-42d8-97da-a7a43f4e461c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>human_readable_id</th>\n",
       "      <th>community</th>\n",
       "      <th>parent</th>\n",
       "      <th>level</th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>full_content</th>\n",
       "      <th>rank</th>\n",
       "      <th>rank_explanation</th>\n",
       "      <th>findings</th>\n",
       "      <th>full_content_json</th>\n",
       "      <th>period</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c81a85511b7246ac8f93a15783e26855</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>Decision Tree Algorithms: ID3, C4.5, and CART</td>\n",
       "      <td>This community revolves around key decision tr...</td>\n",
       "      <td># Decision Tree Algorithms: ID3, C4.5, and CAR...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'ID3 is recognized as a class...</td>\n",
       "      <td>{\\n    \"title\": \"Decision Tree Algorithms: ID3...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0c4efb0094a64a3abd1a184b1baab37e</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>C4.5 and Decision Tree Algorithms</td>\n",
       "      <td>This community is centered around the C4.5 dec...</td>\n",
       "      <td># C4.5 and Decision Tree Algorithms\\n\\nThis co...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'C4.5 represents a significan...</td>\n",
       "      <td>{\\n    \"title\": \"C4.5 and Decision Tree Algori...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>983b4cf9540e4ee7820611ff6f6d33ba</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>NumPy and ML_Basic_Function Community</td>\n",
       "      <td>This community is centered around the NumPy li...</td>\n",
       "      <td># NumPy and ML_Basic_Function Community\\n\\nThi...</td>\n",
       "      <td>7.5</td>\n",
       "      <td>The impact severity rating is high due to the ...</td>\n",
       "      <td>[{'explanation': 'NumPy is identified as a Pyt...</td>\n",
       "      <td>{\\n    \"title\": \"NumPy and ML_Basic_Function C...</td>\n",
       "      <td>2025-01-08</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 id  human_readable_id  community  parent  \\\n",
       "0  c81a85511b7246ac8f93a15783e26855                  0          0      -1   \n",
       "1  0c4efb0094a64a3abd1a184b1baab37e                  1          1      -1   \n",
       "2  983b4cf9540e4ee7820611ff6f6d33ba                  2          2      -1   \n",
       "\n",
       "   level                                          title  \\\n",
       "0      0  Decision Tree Algorithms: ID3, C4.5, and CART   \n",
       "1      0              C4.5 and Decision Tree Algorithms   \n",
       "2      0          NumPy and ML_Basic_Function Community   \n",
       "\n",
       "                                             summary  \\\n",
       "0  This community revolves around key decision tr...   \n",
       "1  This community is centered around the C4.5 dec...   \n",
       "2  This community is centered around the NumPy li...   \n",
       "\n",
       "                                        full_content  rank  \\\n",
       "0  # Decision Tree Algorithms: ID3, C4.5, and CAR...   7.5   \n",
       "1  # C4.5 and Decision Tree Algorithms\\n\\nThis co...   7.5   \n",
       "2  # NumPy and ML_Basic_Function Community\\n\\nThi...   7.5   \n",
       "\n",
       "                                    rank_explanation  \\\n",
       "0  The impact severity rating is high due to the ...   \n",
       "1  The impact severity rating is high due to the ...   \n",
       "2  The impact severity rating is high due to the ...   \n",
       "\n",
       "                                            findings  \\\n",
       "0  [{'explanation': 'ID3 is recognized as a class...   \n",
       "1  [{'explanation': 'C4.5 represents a significan...   \n",
       "2  [{'explanation': 'NumPy is identified as a Pyt...   \n",
       "\n",
       "                                   full_content_json      period  size  \n",
       "0  {\\n    \"title\": \"Decision Tree Algorithms: ID3...  2025-01-08     7  \n",
       "1  {\\n    \"title\": \"C4.5 and Decision Tree Algori...  2025-01-08     4  \n",
       "2  {\\n    \"title\": \"NumPy and ML_Basic_Function C...  2025-01-08     2  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "report_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2a70ca-a0e8-4e58-8bf8-adc775a0c6cd",
   "metadata": {},
   "source": [
    "- 社区报告最终形成的参考文档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "7fd11b0d-9109-4c90-a871-f61ef6ba5ae2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'id|title|occurrence weight|content|rank\\n1|C4.5 and Decision Tree Algorithms|1.0|\"# C4.5 and Decision Tree Algorithms\\n\\nThis community is centered around the C4.5 decision tree algorithm, which is an advanced version of the ID3 algorithm. C4.5 introduces enhancements such as handling continuous variables and employing pruning techniques to prevent overfitting. The community includes related metrics like Information Gain, Information Value, and Gain Ratio, which are integral to the algorithm\\'s functionality. Relationships within the community highlight C4.5\\'s connections to other decision tree algorithms like CART and its reliance on metrics for dataset splits.\\n\\n## C4.5\\'s advancements over ID3\\n\\nC4.5 represents a significant evolution from the ID3 algorithm by introducing the ability to handle continuous variables and employing pruning techniques to prevent overfitting. These enhancements make C4.5 more robust and effective for practical applications, addressing several limitations of its predecessor. [Data: Entities (1)]\\n\\n## Role of Information Gain in C4.5\\n\\nInformation Gain is a critical metric used by C4.5 to measure the reduction in entropy after a dataset split. This metric is foundational in the decision tree construction process, ensuring more accurate dataset partitioning. C4.5 builds on Information Gain by incorporating Information Value to calculate Gain Ratio, further refining the decision-making process. [Data: Entities (11), Relationships (26)]\\n\\n## Importance of Information Value\\n\\nInformation Value is used by C4.5 to correct Information Gain calculations, thereby improving decision tree splits. This metric plays a crucial role in enhancing the algorithm\\'s accuracy and reliability by ensuring that the dataset splits are based on the most informative attributes. [Data: Entities (12), Relationships (24)]\\n\\n## Gain Ratio as a key metric\\n\\nGain Ratio is a pivotal metric in C4.5, calculated by dividing Information Gain by Information Value. It guides the algorithm in evaluating and selecting dataset splits, ensuring that the decision tree is constructed in a manner that maximizes predictive accuracy. [Data: Entities (10), Relationships (25)]\\n\\n## C4.5\\'s relationship with CART\\n\\nC4.5 and CART are both decision tree algorithms that share similarities in handling continuous variables and utilizing Gain Ratio values for dataset splits. However, they differ in their partitioning strategies, with C4.5 relying on information entropy and CART employing cut points. This relationship highlights the diversity in approaches within decision tree algorithms. [Data: Relationships (4)]\"|7.5\\n0|Decision Tree Algorithms: ID3, C4.5, and CART|1.0|\"# Decision Tree Algorithms: ID3, C4.5, and CART\\n\\nThis community revolves around key decision tree algorithms used in machine learning, specifically ID3, C4.5, and CART. ID3 serves as the foundational algorithm, with C4.5 and CART offering improvements and extensions, respectively. These algorithms are interconnected through their evolutionary development and their application in machine learning tasks, particularly in classification and regression. The community also includes entities like SKLEARN, a machine learning library that implements CART, and various dataset columns (AGE, INCOME, STUDENT, CREDIT_RATING) used in calculating information gain and reducing impurity in decision tree models.\\n\\n## ID3 as the foundational decision tree algorithm\\n\\nID3 is recognized as a classic decision tree algorithm primarily used for classification tasks. It employs information entropy to evaluate the purity of splits, focusing on discrete variables. Despite its limitations, such as the tendency to overfit and inability to handle continuous variables directly, ID3 has played a crucial role in the evolution of decision tree-based machine learning techniques. Its foundational principles have been built upon by successors like C4.5 and CART, highlighting its significance in the community. [Data: Entities (0), Relationships (0)]\\n\\n## C4.5\\'s improvements over ID3\\n\\nC4.5 is an improved version of ID3, designed to address several limitations of its predecessor. It introduces features like continuous variable handling, pruning, and improved attribute selection criteria, making it a more robust and versatile algorithm. These enhancements have allowed C4.5 to maintain the core principles of decision tree construction while overcoming the challenges faced by ID3, thereby solidifying its position in the community. [Data: Relationships (0)]\\n\\n## CART\\'s versatility in handling both classification and regression tasks\\n\\nCART (Classification and Regression Trees) extends the capabilities of decision tree algorithms by supporting both classification and regression tasks. It builds binary trees and uses cut points for dataset partitioning, effectively handling continuous variables. CART\\'s methodology has been integrated into popular machine learning libraries like SKLEARN, making it a versatile and widely used algorithm in the field. [Data: Entities (2), Relationships (7)]\\n\\n## SKLEARN\\'s implementation of CART\\n\\nSKLEARN, or Scikit-learn, is a widely used machine learning library that implements the CART algorithm for decision tree modeling. While it does not directly support ID3 and C4.5, SKLEARN\\'s decision tree models are based on CART\\'s robust framework, enabling users to leverage its capabilities for predictive modeling and data analysis. This integration underscores the importance of CART in the community and its influence on machine learning practices. [Data: Entities (3), Relationships (7)]\\n\\n## Role of dataset columns in decision tree modeling\\n\\nDataset columns such as AGE, INCOME, STUDENT, and CREDIT_RATING play a crucial role in decision tree modeling by calculating information gain and reducing impurity. These columns are used in algorithms like ID3 to partition data effectively, highlighting their significance in the decision-making process of machine learning models. [Data: Entities (6, 7, 8, 9), Relationships (10, 11, 12, 13)]\"|7.5\\n2|NumPy and ML_Basic_Function Community|0.25|\"# NumPy and ML_Basic_Function Community\\n\\nThis community is centered around the NumPy library and the ML_Basic_Function module, which are interconnected through their use in numerical computations for machine learning tasks. NumPy, being a fundamental library for numerical computations in Python, plays a crucial role in the functionality of ML_Basic_Function, indicating a significant dependency relationship.\\n\\n## NumPy\\'s foundational role in machine learning\\n\\nNumPy is identified as a Python library essential for numerical computations, including those required in machine learning algorithms such as decision trees. Its widespread use and importance in data science and machine learning projects underscore its foundational role in these fields. [Data: Entities (4)]\\n\\n## ML_Basic_Function\\'s dependency on NumPy\\n\\nML_Basic_Function, a module or library for machine learning tasks, likely utilizes NumPy for numerical computations. This dependency highlights the interconnectedness of tools and libraries within the machine learning ecosystem and the pivotal role of NumPy in enabling complex computations. [Data: Relationships (9)]\\n\\n## The significance of numerical computations in machine learning\\n\\nNumerical computations are at the heart of machine learning algorithms, enabling the processing and analysis of large datasets. The reliance of ML_Basic_Function on NumPy for these computations illustrates the critical importance of efficient and reliable numerical computation libraries in advancing machine learning technologies. [Data: Entities (4, 5), Relationships (9)]\\n\\n## Potential for innovation and development\\n\\nThe relationship between NumPy and ML_Basic_Function suggests a community with a strong foundation for innovation and development in machine learning. The integration of these tools can lead to advancements in algorithm efficiency, accuracy, and the development of new machine learning models. [Data: Relationships (9)]\"|7.5\\n'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.context_text[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ccc857c-a5a0-4ca6-a642-5112830c671b",
   "metadata": {},
   "source": [
    "- 模型调用次数和使用的token数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a9b6f9a1-33e8-41f2-bf77-3dd510813ef1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LLM calls: 2. Prompt tokens: 3501. Output tokens: 1006.\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    f\"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}.\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
